Package: readtagger
ReadtaggerReadtagger is a set of tools for describing the origin of reads in a sequenced genome. It can be used to verify and detect integration events of known, exogenous sequences (such as viruses) or endogenous sequences (such as transposons). In addition, there is the possibility of expanding this to de-novo detection without a priori knowledge of the inserted sequences).Installationpip install readtaggerDescriptionThe readtagger module can be used as a tool for verifying transposable element (TE) insertions predicted by TEMP and TEPID, but it can also serve as the basis for detecting insertions using the findcluster command (see below). The readtagger module examines the origin of sequencing reads by aligning reads to multiple references and recording possible homologies as custom SAM tags. This works for single and paired-end reads, so that one obtains knowledge about a read that is aligned at a physical location in the genome as well as about its mate.The readtagger command evaluates potential homologies, and (by default) discards homologies for alignments that do not improve the nucleotide usage, For example if the unclipped portion of an alignment has a homology with a transposable element, while we only care about the clipped, unaligned portion. Similarly, we discard homologies for aligned pairs that are well within the expected insert size. These precautions reduce noise in regions of the genome that have spurious homologies to transposable elements, as not all of these regions are masked by tools such as RepeatMasker.To generate insertion calls, one can use the findcluster command.findcluster operates in multiple phases, that are roughly:Divide the input file into approximately 100 KB bins. This allows parallelization over multiple CPU cores or machines. Splitting files may introduce artefacts when splitting in a cluster of tags, therefore we attempt to find regions that have no tagged alignments.For each region we determine clusters of tagged alignments. A new cluster is created once additional alignments are not overlapping the current cluster coordinates. This is a rough first-pass that often includes multiple actual insertions and/or separates clusters that belong together, but does summarize all tagged alignments. Additionally we generate clusters of soft-clipped reads (independent of transposable element homology).For each cluster, we ask whether there are any other clusters close by (the search area depends on the insert size distribution) that could potentially be merged because they provide evidence for the same insertion. All reachable clusters are tested by all-against-all alignments using edlib. Putative 5p and 3p sites are determined using a consensus alignment if split read evidence is available.We then ask whether the clusters are internally consistent. This means that alignments that point for example away from the putative insertion site are taken from the cluster and form a new, second cluster.We repeat steps 3 and 4 until the number of clusters at each pass stays constant.We assign soft-clipping clusters to transposable element clusters if any read in the soft clipping cluster is also member of a transposable element cluster. (This is important in a separate step that links transposable element insertion sites across samples).We then quantify the fragments that support a putative insertion (this includes alignments that did not reach the threshold for being tagged), the fragments that point against an insertion and the uninformative reads and we call genotypes and genotype likelihoods based on the evidence. Alignment evidence for an insertion and evidence against an insertion is tagged using separate tags, which allows for visual verification of the quantification. A VCF file describing the insertions, a BAM file containing relevant alignments and a fasta file of assembled sequences is written per 100 KB region.Merge individual filesUsageTo tag reads in filea.bamwith fileb.bamand output to path
output.bam, typereadtagger --tag_file a.bam --annotate_with b.bam ----output_file output.bamThis will by default tag reads with the AD, AR, BD and BR tags, where
the AD tag has detail mapping information for the current read, while
the BD tag has the information for the mate. AR and BR contain the
aligned reference (i.e chromosome). The first letter can be changed on a
per-file basis by appending “:first_letter_read:first_letter_mate”
to the file path. To change the above example into X for the read and Y
for the mate, run:readtagger --tag_file a.bam --annotate_with b.bam:X:Z ----output_file output.bamTo tag one bam file using multiple alignment files, run:readtagger --tag_file a.bam --annotate_with b.bam:A:B c.bam:C:D ----output_file output.bamNow reads that align in fileb.bamwill be tagged with AR, AD and
BR, BD, while reads aligned in filec.bamare marked with CR, CD and
DR, DD.Advanced usageTo see the advanced options, type:readtagger -hTestingIf you modify readtagger, you can run all tests by running tox:pip install tox
toxHistory0.5.25 (2020-04-07)Expose different style, transform Nan to 00.5.24 (2020-04-07)Various updates to plot_coverage tool0.5.23 (2020-03-23)Release a non-broken version, (conda-build problem in 0.5.22)0.5.22 (2020-03-23)Allow one or more alignment files in plot_coverage tool0.5.21 (2020-01-12)Make normalize_readsizes more efficient (again)0.5.20 (2020-01-11)Make normalize_readsizes more efficient for large datasets0.5.19 (2020-01-11)Fix file opening in normalize_readsizes0.5.18 (2019-12-18)Drop recursion from normalize_readsizes, didn’t seem to work0.5.17 (2019-11-05)Add summarize_fragments command
Normalize length of multiple long read fastq files
Update dependencies0.5.16 (2019-09-05)Annotate clusters that are composed exclusively of proper pairs0.5.15 (2019-09-04)Relax cluster merging requirement0.5.14 (2019-09-03)Allow 15 instead of 10 nt between split and TE start0.5.13 (2019-09-03)Fix aligned_segment_corresponds_to_transposable_element0.5.12 (2019-09-02)Write fasta and re-align with bwa after last join_clusters
Test and fix TE overlap annotation0.5.11 (2019-09-01)Annotate overlaps of insertions of the same kind0.5.10 (2019-09-01)Write out matching softclip cluster0.5.9 (2019-08-31)Fix crash when corrected start/end can’t be calculated.0.5.8 (2019-08-31)Improve joining of clusters0.5.7 (2019-08-29)Don’t mark clusters incompatible after splitting them out from original cluster
Use corrected or uncorrected start/end when finding reachable clusters
Also allow joining with downstream reverse cluster
Detect local maximum proper pair size0.5.6 (2019-08-28)Always produce contig fasta, required for refining cluster positions
Allow +/- 10 nucleotides between alignment and insert start/end0.5.5 (2019-08-27)Add timeout to cap3 callUpdate dependencies0.5.4 (2019-07-29)Bump pysam dependency to 0.15.3, contains important fixes0.5.3 (2019-07-28)Sort ‘extract_variants’ output alignment0.5.2 (2019-07-28)Add ‘extract_variants’ tool for extracting insertion evidence from long reads0.5.1 (2019-06-13)Only associate clipping pattern with insertion if pattern matches breakpoint sequence0.5.0 (2019-06-12)Drop support for Python 20.4.20 (2019-06-12)Keep all associated softclip patterns when merging adjacent read clusters0.4.19 (2019-02-15)Fix findcluster crash when reference contains colon.0.4.18 (2019-02-14)Use logger.warning instead of deprecated logger.warnDrop now unused qname_cmp_funcFix alignment splitting, fixes untagged reads and speed issues0.4.17 (2019-02-10)Fix a bug that would lead to wrong chunk sizes0.4.16 (2019-01-28)Drop samtools, do everything via pysam0.4.15 (2019-01-15)Add missing samtools dependency0.4.14 (2019-01-15)Build Conda package for python 3 only0.4.13 (2019-01-14)Update pinned dependenciesFix travis deployment0.4.12 (2018-08-21)Allow multiple inputs to readtaggerAllow passing multiple control files to confirm_insertions scriptFix matching of short 3p clipped sequences0.4.11 (2018-05-18)Add a script that merges findlcuster VCF outputAllow 5 nt overlaps at cluster consistency checkInclude VALID_TSD in INFO field and write out PE supportSort output VCF fileGenerate IDs using reference_name start and cluster orderImprove support for arbitrary insertion names0.4.10 (2018-03-30)Include unmapped but tagged mates in veriefied tagsUpdate findcluster galaxy tool and fix softclip cluster idsUse a unique ID as variant IDStop collection evidence once we reach 10000 readsSpeed up finding of soft clip clustersImplement VCF outputMake loglevel configurable for findcluster script and add option to output log to fileLook for softclipped reads in a 15nt window and compare 5p clips by their endAdd script and tool to confirm/reject insertionsRefine the detection of TE clusters that are very close to each otherVerify that reads really support a specific insertionFix sorting to CRAM outputMove sorting of softclip clusters to merging phaseSkip finding softclipped clusters when skipping TE clustersAnnotate softclips as part of TEsEmbedd SoftClipClusterFinder in ClusterFinderFix softclipped positions when read contains deletions0.4.9 (2018-01-23)Fix deployment to PyPI0.4.8 (2018-01-23)Update test data output and allow:Add edlib to requirements in setup.pyAdd softclip finder testBuild on python-3.6Make futures library conditional for python2Drop temporary from requirementsExtend testcoverageDropexternal_binfrom BamAlignmentWriterMany small simplifications, bugfixes and enhaced testsImprove reporting of 5p and 3p clipsAdd some wigglespace for finding the most likely TSD positionKeep insertions associated with deletion intactAdd testcase for a cluster that should not be splitFix if/else logic for genotypesSkip “genomic sinks” with lots of TE evidenceContinue on RuntimeErrorImprove splitting of input fileNeed tofetchreads in the specified region if usingexternal_bin=FalseDon’t use external samtools when finding clustersIdentify decoy regions based on cluster densityDrop reraise_with_stack, doesn’t work on py3Fix outdated min/max coordinates leading to dropped chunksRe-raise any exceptions when processing chunksFix OrderedDict syntax for py2 compatibilityImprove logging when splitting input into chunksDon’t remove read that isn’t present anymoreFix return value when assembling too many readsFix limiting of region when using multiple threadsReport maximum MAPQ of read evidence for a clusterBump minimum MAPQ to 4 by default and make it configurableRefactor cap3 assembly (so it can be exchanged more easily) and add limit to how many reads it will assembleFix and apply read_is_compatible to all read with BD tagGeneralize marking clusters as compatible or incompatible and apply at every cluster split or joinEstimate nref/nalt using overlap of start and end if start and end are more than 50nt apartSkip clusters of reads that are inconsistentRemove redundant parenthesis, fix typoAllow non-proper pairs when counting evidenceAccount for max. mate distance when joining clusterAdd new dependencies to conda recipePrevent joining clusters that we previously split explicitlyDon’t thread/cache joining of clusterUse lru_cache for some cigar operationsUse cigar_to_max function consistentlyMake use of new AlignmentHeader object (old method now very slow)Use edlib align instead of Cap3AssemblyFix evidence_against functionalityOutput reads that count as non-supportAllow picking up location of reference_fasta via env var for quicker test executionFix 3p evidence bam, fix nref with 1 breakpointUpdate test-dataAssign left/right based on AD if AD and BD are setMake counting more accurate, cleanup various Cluster counts and write out split reads found viaevidence_for_five/three_pCollect evidence for insertionsFix a typo inget_breakpoint_sequenceFix resolving consensus ties if tie containsNUpgrade to pysam 0.14Make split_ads a property since the splits can updateFix typo in dumb_consensus helpAdd IUPAC to nucleotides dictRestructure non_evidence so that evidence for and against can be countedUsereference_startinstead of deprecatedposImplementget_breakpoint_sequenceas a method of TargetSiteDuplicationAddevidence_forfunctionUpdate planemo from 0.46.1 to 0.48.0Refine the cluster merging logicFix the overlap calculation, in case the re-aligned contig ends up at a different positionUpdate test data output, genotype outputs with higher precision (sigh)0.4.7 (2018-01-23)Fix Exception that occurs when cluster doesn’t have an associated contigFix TE alignment logic when using pre-indexed transposon referencesControl which reads extend a cluster during cluster refinementAdd a safeguard to avoid merging unrelated, far-away clusters0.4.6 (2017-12-13)Deploy to conda on py3 as wellMake sure cluster chunks are orderedAvoid hangs due to expection in multiprocessing tasks0.4.2 (2017-12-13)Fix passing of region specification to pileup enginePoint out typical useage of –reference_fasta and –reference_indexFix cheetah bwa index variable for findcluster galaxy tool0.4.1 (2017-11-20)Add matplotlib and pandas to dependenciesAdd a script that can plot coverage as an area plot between two bam filesUpdate dependenciesIf either three_p or five_p of a tsd is unknown assign the available use the available side to test of a read belongs to the left or right side of an insertionFix crash for unaligned(?) readsChange deprecacted alen, pos and mpos to current replacementsTune clusterfinding for misaligned long reads0.4.0 (2017-11-09)Fixes for CRAM input and outputAdjust chunk-size in readtagger based on readlength (for pacbio/nanopore reads)Cleanup temporary bwa indexesDependency updates0.3.25 (2017-06-21)Refine cluster coordinates using an Assembly strategyFix GFF sorting on python 3Improve BWA alignment settings (default to intractg plus -Y) and add align_contigs method to SimpleAlignerAdd pysamtools_view commandImprove cluster-splittingAdd multiprocessing-logging recipeOnly output BWA stderr if the exit code is not zeroAdd a function to sort gff filesClose open file descriptorsMake imprecise insertion sites more realisticFix read_index propertyAdapt readtagger to higher coverage datasetsFix readtagger crash when not producing discard tag file.Add number of mates for left and right support to GFFSplit clusters that start with reverse reads conatining only BD tags0.3.24 (2017-05-11)Split cluster if there are multiple polarity switches between Forward and Reverse orientationManipulate copy of cigarlist to avoid numpy issue0.3.23 (2017-05-09)Expose reference fasta option in bam_readtagger.xml0.3.22 (2017-05-09)Move readtagger CLI form argparse to clickIndex bamfile if neccesaryReplace multipocessing pool with ProcessPoolExecutorSet the matesequence while tagging readsFix false positives in readtagger moduleDo cap3 assembly in shared memory if passing –shm_dir or if SHM_DIR environment variable is definedParallelize findlcluster by splitting input bamAdd check_call.py script for rapidly verifying IGV screenshots0.3.21 (2017-04-27)Fix crash when determining reference name0.3.20 (2017-04-27)Guess the best TE match and write it into GFF ParentFix case where input files are already sortedRemove blast from requirements0.3.19 (2017-04-27)Skip creating tempdirs in current working directoryRemove blast-specific filesSwitch to using BWA for annotating detected insertionsAdd more logging and default to not changing sort order unless specifically demandedDo dovetailing on coordinate-sorted file0.3.18 (2017-04-25)Fix small outputs due to switching of-tand-aoptions0.3.17 (2017-04-25)Fix file seekingUpdate dependencies0.3.16 (2017-04-23)Parallelize readtagger0.3.15 (2017-04-20)Do not count reads as support if both AD and BD tag contribute to an insertionRemove sambamba support0.3.14 (2017-04-19)Perform readtagging on readname sorted files.Catch possible errorsAdd BWA alignment module to replace Blast0.3.13 (2017-04-05)Add possibility to output cluster contigs as fasta0.3.12 (2017-03-31)Fix and accelerate the calculation of nref (=non support evidence)Update priors and genotype frequrencies to a more realistic model0.3.11 (2017-03-28)Add a testcase for genotyping moduleStream over full alignment file instead of fetching regions,
pysam.AlignmentFile.fetch is too slow0.3.10 (2017-03-26)Revert local conda dependency resolutionFix readtagger.add_mate to work also if one mate is unmapped0.3.9 (2017-03-26)Add a genotyping moduleKeep tags for alternative alignments if mates are not in a proper pair0.3.4 (2017-03-02)Speed up assembly steps using multithreadingImplement a cache for the Cluster.can_join method0.3.3 (2017-03-02)Fix a crash when writing GFF for a cluster of hardclipped readsChange confusing variable names and copypasted docstring0.3.2 (2017-03-02)Fix another crash when tuple starts with 1,2,7 or 80.3.1 (2017-03-02)Fix a crash when a mismatch is the last item in a cigartuple0.3.0 (2017-03-02)Add a galaxy tool for the findcluster scriptAdd new script that finds clusters of reads and outputs GFF or BAM files with these clusters.Implement writing clusters as GFF filesImplement writing out reads with cluster number annotated in CD tag.Implement merging of clusters based on whether reads contribute to common contigsUse cached-property where it makes senseAdd module to find, join and annotate clusters of readsRepresent cigartuple as namedtupleAdd a Roadmap fileAdd more logic for finding ends of insertions andManipulate cluster of reads to find TSDsAdd module for cap3 assembly and manipulation of assembled readsFix conda recipe script entrypoints0.2.0 (2017-02-21)Reformat help text in galaxy wrappersAdd add_matesequence script to add the sequence of the mate of the current read as a tagAdd option to discard alternative tag if read is a proper pairStitch cigars that are separated by I or D eventsAdd a tag tuple that knows how to format itselfUpdate README.rst example with current default tag prefixTest with and without discarding verified readsSymlink test-files that are shared with the galaxy test, add testcase for allow_dovetailing scriptFix HISTORY.rst formatting0.1.13(2017-02-17)Add instructions for developmentInstall planemo in deployment step0.1.12(2017-02-17)Test deployment again0.1.11 (2017-02-17)Test deployment0.1.10 (2017-02-17)Fix toolshed deployment0.1.9 (2017-02-17)Add automated deployment to Galaxy ToolshedAdd instructions for development and release process0.1.8 (2017-02-17)Minor release to test release process0.1.7 (2017-02-17)Extend testing with coverage testingAutomate deployment to pypi and condaRegister project with pyup.io0.1.6 (2017-02-16)Rename to readtaggerFix bug with stdin closing file descriptor too early, leading to corrupt
BAM filesExtend testing0.1.5 (2017-02-12)Add option (-wd) to write suboptimal tag into separate BAM fileAdd option (-wv) to write verified tags into separate BAM filePerformance improvments by letting sambamba handle BAM reading
and writing. Also elimininate regualr expression to parse cigarstring0.1.4 (2017-02-10)Add option (-k) to keep alternative tags if they do not
explain the softclipped read any better.
Default is to discard them.0.1.3.2 (2017-02-08)Fix dovetailing script0.1.3 (2017-02-07)Add option to allow dovetailing in alignment files when tagging readsAdd separate entrypoint for standalone script0.1.2 (2017-02-05)Add conda recipePython3 string fix0.1.0 (2017-02-05)Initial version