Package: ribodetector
RiboDetector - Accurate and rapid RiboRNA sequences Detector based on deep learningAbout RibodetectorRiboDetectoris a software developed to accurately yet rapidly detect and remove rRNA sequences from metagenomeic, metatranscriptomic, and ncRNA sequencing data. It was developed based on LSTMs and optimized for both GPU and CPU usage to achieve a10times on CPU and50times on a consumer GPU faster runtime compared to the current state-of-the-art software. Moreover, it is very accurate, with ~10times fewer false classifications. Finally, it has a low level of bias towards any GO functional groups.Prerequirements1. Createcondaenv and installPython v3.8To be able to useRiboDetector, you need to installPython v3.8orv3.9(make sure you have version3.8because3.7cannot serialize a string larger than 4GiB) withconda:condacreate-nribodetectorpython=3.8
condaactivateribodetector2. Installpytorchin the ribodetector env if GPU is availableTo installpytorchcompatible with your CUDA version, please fellow this instruction:https://pytorch.org/get-started/locally/. Our code was tested withpytorch v1.7,v1.7.1,v1.10.2.Note: you can skip this step if you don't use GPUInstallationUsing pippipinstallribodetectorUsing condacondainstall-cbiocondaribodetectorUsageGPU modeExampleribodetector-t20\-l100\-iinputs/reads.1.fq.gzinputs/reads.2.fq.gz\-m10\-errna\--chunk_size256\-ooutputs/reads.nonrrna.1.fqoutputs/reads.nonrrna.2.fqThe command lind above excutes ribodetector for paired-end reads with mean length 100 using GPU and 20 CPU cores. The input reads do not need to be same length. RiboDetector supports reads with variable length. Setting-lto the mean read length is recommended.Full helpusage:ribodetector[-h][-cCONFIG][-dDEVICEID]-lLEN-i[INPUT[INPUT...]]-o[OUTPUT[OUTPUT...]][-r[RRNA[RRNA...]]][-e{rrna,norrna,both,none}][-tTHREADS][-mMEMORY][--chunk_sizeCHUNK_SIZE][-v]rRNAsequencedetector

optionalarguments:-h,--helpshowthishelpmessageandexit-cCONFIG,--configCONFIGPathofconfigfile-dDEVICEID,--deviceidDEVICEIDIndicesofGPUstoenable.Quotatedcomma-separateddeviceIDnumbers.(default:all)-lLEN,--lenLENSequencingreadlength(meanlength).Note:theaccuracyreducesforreadsshorterthan40.-i[INPUT[INPUT...]],--input[INPUT[INPUT...]]Pathofinputsequencefiles(fastaandfastq),thesecondfilewillbeconsideredassecondendiftwofilesgiven.-o[OUTPUT[OUTPUT...]],--output[OUTPUT[OUTPUT...]]PathoftheoutputsequencefilesafterrRNAsremoval(samenumberoffilesasinput).(Note:2timesslowertowritegzfiles)-r[RRNA[RRNA...]],--rrna[RRNA[RRNA...]]PathoftheoutputsequencefileofdetectedrRNAs(samenumberoffilesasinput)-e{rrna,norrna,both,none},--ensure{rrna,norrna,both,none}Ensurewhichclassificaionhashighconfidenceforpairedendreads.norrna:outputonlyhighconfidentnon-rRNAs,therestareclasifiedasrRNAs;rrna:viceversa,onlyhighconfidentrRNAsareclassifiedasrRNAandtherestoutputasnon-rRNAs;both:bothnon-rRNAandrRNApredictionwithhighconfidence;none:givelabelbasedonthemeanprobabilityofreadpair.(Onlyapplicableforpairedendreads,discardthereadpairwhentheirpredicitonsarediscordant)-tTHREADS,--threadsTHREADSnumberofthreadstouse.(default:10)-mMEMORY,--memoryMEMORYAmount(GB)ofGPURAM.(default:12)--chunk_sizeCHUNK_SIZEUsethisparameterwhenhavinglowmemory.Parsingthefileinchunks.NotneededwhenfreeRAM>=5*your_file_size(uncompressed,sumofpairedends).Whenchunk_size=256,memory=16itwillload256*16*1024readseachchunk(use~20GBfor100bppairedend).--logLOGLogfilename-v,--versionShowprogram'sversionnumberandexitCPU modeExampleribodetector_cpu-t20\-l100\-iinputs/reads.1.fq.gzinputs/reads.2.fq.gz\-errna\--chunk_size256\-ooutputs/reads.nonrrna.1.fqoutputs/reads.nonrrna.2.fqThe above command line excutes ribodetector for paired-end reads with mean length 100 using 20 CPU cores. The input reads do not need to be same length. RiboDetector supports reads with variable length. Setting-lto the mean read length is recommended. If you need to save the log into a file, you can specify it with--log <logfile>Note: when usingSLURMjob submission system, you need to specify--cpus-per-taskto the number you CPU cores you need and set--threads-per-coreto 1.Full helpusage:ribodetector_cpu[-h][-cCONFIG]-lLEN-i[INPUT[INPUT...]]-o[OUTPUT[OUTPUT...]][-r[RRNA[RRNA...]]][-e{rrna,norrna,both,none}][-tTHREADS][--chunk_sizeCHUNK_SIZE][-v]rRNAsequencedetector

optionalarguments:-h,--helpshowthishelpmessageandexit-cCONFIG,--configCONFIGPathofconfigfile-lLEN,--lenLENSequencingreadlength(meanlength).Note:theaccuracyreducesforreadsshorterthan40.-i[INPUT[INPUT...]],--input[INPUT[INPUT...]]Pathofinputsequencefiles(fastaandfastq),thesecondfilewillbeconsideredassecondendiftwofilesgiven.-o[OUTPUT[OUTPUT...]],--output[OUTPUT[OUTPUT...]]PathoftheoutputsequencefilesafterrRNAsremoval(samenumberoffilesasinput).(Note:2timesslowertowritegzfiles)-r[RRNA[RRNA...]],--rrna[RRNA[RRNA...]]PathoftheoutputsequencefileofdetectedrRNAs(samenumberoffilesasinput)-e{rrna,norrna,both,none},--ensure{rrna,norrna,both,none}Ensurewhichclassificaionhashighconfidenceforpairedendreads.norrna:outputonlyhighconfidentnon-rRNAs,therestareclasifiedasrRNAs;rrna:viceversa,onlyhighconfidentrRNAsareclassifiedasrRNAandtherestoutputasnon-rRNAs;both:bothnon-rRNAandrRNApredictionwithhighconfidence;none:givelabelbasedonthemeanprobabilityofreadpair.(Onlyapplicableforpairedendreads,discardthereadpairwhentheirpredicitonsarediscordant)-tTHREADS,--threadsTHREADSnumberofthreadstouse.(default:20)--chunk_sizeCHUNK_SIZEchunk_size*1024readstoloadeachtime.Whenchunk_size=1000andthreads=20,consumming~20Gmemory,bettertobemultiplesofthenumberofthreads..--logLOGLogfilename-v,--versionShowprogram'sversionnumberandexitNote: RiboDetector uses multiprocessing with shared memory, thus the memory use of a single process indicated inhtoportopis actually the total memory used by RiboDector. Some job submission system like SGE mis-calculated the total memory use by adding up the memory use of all process. If you see this do not worry it will cause out of memory issue.FAQWhat should I set for-lwhen I have reads with variable length?You can set the-lparameter to the mean read length if you have reads with variable length. The mean read length can be computed withseqkit stats. This parameter tells how many bases will be used to capture the sequences patterns for classification.How does-eparameter work? What should I set (rrna,norrna,none,both)?This parameter is only necessary for paired end reads. When setting torrna, the paired read ends will be predicted as rRNA only if both ends were classified as rRNA. If you want to identify or remove rRNAs with high confidence, you should set it torrna. Conversely,norrnawill predict the read pair as nonrRNA only if both ends were classified as nonrRNA. This setting will only output nonrRNAs with high confidence.bothwill discard the read pairs with two ends classified inconsistently, only pairs with concordant prediction will be reported in the corresponding output.nonewill take the mean of the probabilities of both ends and decide the final prediction. This is also the default setting.I have very large input file but limited memory, what should I do?You can set the--chunk_sizeparameter which specifies how many reads the software load into memory once.What should I do if RiboDetector hangs with SLURM?The most likely cause is that the requested computational resource is not sufficient for the input file. You need to make sure you specified--cpus-per-taskto the number you CPU cores you want to use and set--threads-per-coreto 1 in the SLURM submission script or command. If the issue remains, you can try to reduce the memory use by setting--chunk_sizeparameter inribodetectororribodetector_cpucommand.CitationDeng ZL, Münch PC, Mreches R, McHardy AC. Rapid and accurate detection of ribosomal RNA sequences using deep learning.Nucleic Acids Research. 2022. (https://doi.org/10.1093/nar/gkac112)AcknowledgementsThe scripts from thebasedir were from the templatepytorch-templatebyVictor Huangand othercontributors.