opened
charlesdavid
username_0
Hi Alex, 
I am attempting to use the 2-pass method on my data, but STAR is crashing with a seg fault:
The input files were verified correct format, no errors.
STAR reads in the genome, performs the first pass, but crashes when trying to run the second pass.
I have also tried to manually feed in the SJs running STAR the second time, but still crashes.
There is only a header in the `Log.progress.out` and `Log.out` ends with `Created thread # 1-31` as I am using 32 threads. FYI, this happens for both versions 2.5.2b and 2.5.3a.
Here is the error message:

`/home/cflcyd/.lsbatch/1510019549.586204: line 8: 37165 Segmentation fault      (core dumped) STAR --runThreadN 32 --genomeDir 007.STAR/index --readFilesIn Test_sortmerna_trimmomatic_MIA_R1.fastq Test_sortmerna_trimmomatic_MIA_R2.fastq --chimSegmentMin 15 --chimJunctionOverhangMin 15 --alignMatesGapMax 20000 --alignIntronMax 20000 --outSAMtype BAM Unsorted --outSAMprimaryFlag AllBestScore --outSAMstrandField intronMotif --sjdbFileChrStartEnd Test_sortmerna_trimmomatic_MIA_SJ.out.tab --outFileNamePrefix TEST_ --outQSconversionAdd -31`
STAR Two-Pass Mode Not Working: Seg Fault
issue
created
alexdobin
username_1
Hi Charles,

could you please send me the Log.out file?

Cheers
Alex
comment
created
charlesdavid
username_0
Hi Alex,

Here it is.
[TEST_Log.out.txt](https://github.com/username_1/STAR/files/1465274/TEST_Log.out.txt)
comment
created
alexdobin
username_1
Hi Charles,

nothing suspicious in the Log.out file.
To narrow down the problem, Could you please try the following:
1. Map with default parameters, only set the --genomeDir and --readFilesIn
2. Map a few thousand reads and map them.
3. Generate a genome for just one chromosome and map a few reads to it.

If none of the above works, could you send me the one chromosome and a few reads that still cause the seg-fault for further testing.

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

I also have a problem with the two-pass mode (STAR 2.7.0e).

I get a core dump after mapping > 100 mio reads in the 2nd round. In the first round it's all fine. Happens with the option --twopassMode Basic as well as if I do it manually (in both cases only in the 2nd round). I tried once to remove the SJs at the ends of the chromosome (closer than 250 bp) but this did not help. I could narrow it down to 150'000 reads. 

[Here is a link to all the files that I used for the test below:](https://www.dropbox.com/s/a19hvnyd8ucn8ov/forAlex.tar.gz?dl=1)

```
TESTFOLDER="$HOME/temp"
SAMPLE="Bb-tachyzoites"
STARIDXPASS="$TESTFOLDER/2ndPassIndex"
INPUTSEQ="$TESTFOLDER/withStructAndSmallVar.fasta"
SJTABLE="$TESTFOLDER/${SAMPLE}SJ.out.tab"
FORWARD="$TESTFOLDER/${SAMPLE}_R1_bugTest_select.tr.fq"
REVERSE="$TESTFOLDER/${SAMPLE}_R2_bugTest_select.tr.fq"
OUTPREFIX="$TESTFOLDER/${SAMPLE}_2ndPass"

STAR --runThreadN 4 --runMode genomeGenerate --genomeDir $STARIDXPASS --sjdbFileChrStartEnd $SJTABLE --sjdbOverhang 149 --genomeFastaFiles $INPUTSEQ

STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD $REVERSE --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted

# returns:
Mar 19 15:22:43 ..... started STAR run
Mar 19 15:22:43 ..... loading genome
Mar 19 15:22:44 ..... started mapping
Segmentation fault (core dumped)
```

What seems very odd is that I cannot narrow it down any further. If I remove the first half of the reads it does not crash. If I remove the second half it does not crash either. It only crashes using all of the remaining 150'000 reads.

```
wc -l $FORWARD

head -n 300000 $FORWARD > $FORWARD.firstHalf
head -n 300000 $REVERSE > $REVERSE.firstHalf
STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD.firstHalf $REVERSE.firstHalf --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted

# no crash, finishes as usual

tail -n 300000 $FORWARD > $FORWARD.secondHalf
tail -n 300000 $REVERSE > $REVERSE.secondHalf
STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD.secondHalf $REVERSE.secondHalf --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted

# no crash, finishes as usual
```

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

could you please try the latest patch on GitHub [master](https://github.com/username_1/STAR/archive/master.zip).
I have fixed a bug with 2-pass, it  may resolve your issue. if not I will look into it.

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

Thanks. I downloaded the archive from the link and I used the binary in "STAR-master/bin/Linux_x86_64_static/". Still the same behavior.

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

thanks for trying it - I will look into it closely now.

Cheers
Alex
comment
created
alexdobin
username_1
Hi Marc,

this looks like an issue with a small genome rather than 2-pass: you need to scale down --genomeSAindexNbases when generating the genome, for both 1st and 2nd pass (even if there were no problem in the 1st pass).

--genomeSAindexNbases 12 worked with your fastq, but I would go lower to 10 just to be safe.

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

Thanks for the suggestion. I tried --genomeSAindexNbases 12 with the version from the master and 2.7.0e. Both crash (two different machines by the way - Ubuntu Server 16.04 on Intel and Kubuntu 18.04 on AMD). Even if I go down to --genomeSAindexNbases 10 or --genomeSAindexNbases 6 it crashes (I double-checked the genomeParameters.txt). It still only happens if I use all reads and not if I split the fastqs further.

On which OS did you test it?

It seems to be related at least partly to the splice junctions as it only happens if I insert them. I tried now to remove all junctions at the contig ends (first and last 1 kb this time) and all junctions on contigs shorter than 1 Mb. Still crashes.

Any other ideas?

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

do the crashes happen on the same small dataset as before?
Also, could you try the pre-compiled executables from the bin/ ?
My systems is CentOS-7.

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

Yes, same small dataset. 

I used only precompiled binaries so far. The 2.7.0e is from miniconda and the other one from the link you posted previously (master.zip). 

I tested it briefly with a CentOS-7.6.1810. Crashed as well. The same test as at the beginning with the precompiled executable from the master.zip:

```
TESTFOLDER="$HOME/temp"
SAMPLE="Bb-tachyzoites"
STARIDXPASS="$TESTFOLDER/2ndPassIndex"
INPUTSEQ="$TESTFOLDER/withStructAndSmallVar.fasta"
SJTABLE="$TESTFOLDER/${SAMPLE}SJ.out.tab"
FORWARD="$TESTFOLDER/${SAMPLE}_R1_bugTest_select.tr.fq"
REVERSE="$TESTFOLDER/${SAMPLE}_R2_bugTest_select.tr.fq"
OUTPREFIX="$TESTFOLDER/${SAMPLE}_2ndPass"

STAR --runThreadN 4 --runMode genomeGenerate --genomeSAindexNbases 10 --genomeDir $STARIDXPASS --sjdbFileChrStartEnd $SJTABLE --sjdbOverhang 149 --genomeFastaFiles $INPUTSEQ

STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD $REVERSE --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted

# segmentation fault
```

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

I am running it with exactly the same parameters and it runs fine...
I will run it with valgrind to see if there any hidden problems.
On your side, could you try to compile on your machine - maybe there is some kind of incompatibility there. And then please send me the Log.out of a failed run.

Thanks!
Alex
comment
created
MWSchmid
username_2
Hi Alex

Here is the log from genomeGenerate:

[Log.out.zip](https://github.com/username_1/STAR/files/3002091/Log.out.zip)

And here from alignReads (precompiled executable from the master.zip, static linkage):

[Bb-tachyzoites_2ndPassLog.out.zip](https://github.com/username_1/STAR/files/3002142/Bb-tachyzoites_2ndPassLog.out.zip)

Then I also tried compiling it myself:

```
git clone https://github.com/username_1/STAR
cd STAR/source
make -j 4 STAR
cd

TESTFOLDER="$HOME/temp"
SAMPLE="Bb-tachyzoites"
STARIDXPASS="$TESTFOLDER/2ndPassIndex"
INPUTSEQ="$TESTFOLDER/withStructAndSmallVar.fasta"
SJTABLE="$TESTFOLDER/${SAMPLE}SJ.out.tab"
FORWARD="$TESTFOLDER/${SAMPLE}_R1_bugTest_select.tr.fq"
REVERSE="$TESTFOLDER/${SAMPLE}_R2_bugTest_select.tr.fq"
OUTPREFIX="$TESTFOLDER/${SAMPLE}_2ndPass"

$HOME/STAR/source/STAR --runThreadN 4 --runMode genomeGenerate --genomeSAindexNbases 10 --genomeDir $STARIDXPASS --sjdbFileChrStartEnd $SJTABLE --sjdbOverhang 149 --genomeFastaFiles $INPUTSEQ

$HOME/STAR/source/STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD $REVERSE --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted
```

Does not work, crashes as well. Here is the log:

[self_compiled_Bb-tachyzoites_2ndPassLog.out.zip](https://github.com/username_1/STAR/files/3002160/self_compiled_Bb-tachyzoites_2ndPassLog.out.zip)

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

nothing suspicious in the Log files, and valgrind did not catch any relevant problems.
Could you download the genome I generated and try mapping to it?
It will tell us whether the error is in the genome generation or mapping.

Thanks!
Alex
comment
created
alexdobin
username_1
Sorry, forgot the link:
http://labshare.cshl.edu/shares/gingeraslab/www-data/dobin/STAR/Tests/Marc/
comment
created
MWSchmid
username_2
Hi Alex

I tested it with your genome index. Still crashes. The log looks similar:

[Bb-tachyzoites_2ndPassLog.out.zip](https://github.com/username_1/STAR/files/3007121/Bb-tachyzoites_2ndPassLog.out.zip)

I also tested running it on a single thread (crash) and typing the entire command again without any variables (to double check that there are no odd characters, still crash, can't see anything special with the hex editor either).

Are environment variables or locales important/used?

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

STAR does not use any env variables or locales...
So the problem seems to be with the mapping itself, not genome generation.

I think you would have to try to debug it on your side since I cannot reproduce the problem.
If you are willing to do it, here are the steps:
$ cd source
$ make clean
$ make gdb
$ gdb --args $HOME/STAR/source/STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir $STARIDXPASS --readFilesIn $FORWARD $REVERSE --outFileNamePrefix $OUTPREFIX --outSAMtype BAM Unsorted

and inside gdb:
(gdb) r
The code will run and we should be able to see where seg-fault happens.

Thanks!
Alex
comment
created
MWSchmid
username_2
Hi Alex

Thanks for the suggestion. I'm out of office tomorrow but I will test it on Thursday.

Best regards,

Marc
comment
created
MWSchmid
username_2
Hi Alex

I tested it. It runs perfectly fine with the debug build. Does not matter if in gdb or directly in the console. I verified the crash of the release build again by doing a quick "make clean" and "make STAR" again. Still crashes.

Well, I will just use the debug build for now.

Best regards,

Marc

Here's the output from the debugger:

(gdb) r
Starting program: /home/marc/STAR/source/STAR --runThreadN 4 --runMode alignReads --limitBAMsortRAM 20000000000 --genomeDir /home/marc/temp/2ndPassIndex --readFilesIn /home/marc/temp/Bb-tachyzoites_R1_bugTest_select.tr.fq /home/marc/temp/Bb-tachyzoites_R2_bugTest_select.tr.fq --outFileNamePrefix /home/marc/temp/Bb-tachyzoites_2ndPass --outSAMtype BAM Unsorted
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Mar 28 08:43:16 ..... started STAR run
Mar 28 08:43:16 ..... loading genome
Mar 28 08:43:16 ..... started mapping
[New Thread 0x7fff906d9700 (LWP 5570)]
[New Thread 0x7fff8fed8700 (LWP 5571)]
[New Thread 0x7fff8f6d7700 (LWP 5572)]
[Thread 0x7fff8f6d7700 (LWP 5572) exited]
[Thread 0x7fff8fed8700 (LWP 5571) exited]
[Thread 0x7fff906d9700 (LWP 5570) exited]
Mar 28 08:55:36 ..... finished mapping
Mar 28 08:55:36 ..... finished successfully
[Inferior 1 (process 5566) exited normally]
comment
created
alexdobin
username_1
Hi Marc,

thanks for the tests! So it's a Heisenbug... which are hard to debug.
I will try to run valgrind withoutdebug compilation, maybe it will give us a hint.

Debug compiling is done with -O0, i.e. without any optimizations, which makes the code ~3 times slower.
Please try to compile with (after make clean) and let me know if it crashes or not:
$ make CXXFLAGS="-O2 -pipe -std=c++11 -Wall -Wextra -fopenmp"
and if it still crashes with -O1
These should be faster - if they work.

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

I tried

```
cd STAR/source
make clean
make CXXFLAGS="-O2 -pipe -std=c++11 -Wall -Wextra -fopenmp"
# also tried with STAR at after the CXXFLAGS:
# make CXXFLAGS="-O2 -pipe -std=c++11 -Wall -Wextra -fopenmp" STAR
```

and I got:

g++: warning: parametersDefault.xxd: linker input file unused because linking not done
g++: warning: htslib: linker input file unused because linking not done
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp ParametersSolo.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloRead.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloRead_record.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloReadBarcode.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloReadBarcode_getCBandUMI.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloReadFeature.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloReadFeature_record.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloReadFeature_inputRecords.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp Solo.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloFeature.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloFeature_collapseUMI.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloFeature_outputResults.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SoloFeature_processRecords.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp ReadAlign_outputAlignments.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp ReadAlign.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp STAR.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SharedMemory.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp PackedArray.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp SuffixArrayFuns.cpp
g++ -c  -O2 -pipe -std=c++11 -Wall -Wextra -fopenmp Parameters.cpp
Parameters.cpp: In member function ‘void Parameters::inputParameters(int, char**)’:
Parameters.cpp:340:62: error: ‘COMPILATION_TIME_PLACE’ was not declared in this scope
     inOut->logMain << "STAR compilation time,server,dir=" << COMPILATION_TIME_PLACE << "\n"<<flush;
                                                              ^~~~~~~~~~~~~~~~~~~~~~
Makefile:66: recipe for target 'Parameters.o' failed
make: *** [Parameters.o] Error 1

Best regards,

Marc
comment
created
alexdobin
username_1
Sorry, it should be
$ make CXXFLAGS="-O1 -pipe -std=c++11 -Wall -Wextra -fopenmp -D'COMPILATION_TIME_PLACE=1'"
comment
created
MWSchmid
username_2
Hi Alex

Ok, that worked for the compilation. Though, both, -O2 and -O1, crash.

Let me know if there is something else I can test.

Best regards,

Marc
comment
created
alexdobin
username_1
Hi Marc,

thanks a lot for the tests!
I run valgrind with optimized code, and it did catch a problem, which may (fingers crossed) have caused the seg-fault you observed. 
Please try the code on GitHub master:
https://github.com/username_1/STAR/archive/master.zip
Either pre-compiled executables or compiled with simple "make".

Cheers
Alex
comment
created
MWSchmid
username_2
Hi Alex

Thanks for the fix. I compiled the new code and ran it as always. No crash this time. Seems to be fixed :)

Best regards,

Marc
comment
created
MWSchmid
username_2
Hi Alex

Thanks for the fix. I compiled the new code and ran it as always. No crash this time. Seems to be fixed :)

Best regards,

Marc
comment
closed
alexdobin
username_1
issue
created
alexdobin
username_1
Hi Marc,

thank you very much for your help with debugging it.
I would not be able to do it without your prompt tests.

Cheers
Alex
comment