Package: klumpy
Klumpy: A tool to evaluate the integrity of long-read genome assemblies and illusive sequence motifsInitially developed to verify the Antifreeze glycoproteins in the genomes of two icefishes (seeRivera-Colón, 2023),klumpynow has expanded to include a suite of tools for assessing whether a particular region in a genome assembly is missassembled by leveraging the original data (i.e., raw reads) used in the assembly process, or it can instead be used to search for a gene of interest in a set of sequences. Complete overview of the software and the methods used in the paper can be viewed athere.Installationpython3 -m pip install klumpyComponentsklumpycurrently is composed of 9 different subprograms, two are which are the primary subprograms, two which can illustrate the results, with the remaining 5 designed to supplement the four other subprograms. Depending on whether you have a gene of interest, as in the case with the Icefishes and AFGPs, you can start the analysis withfind_klumps. If the aim is to assess the quality of your reference genome, thescan_alignmentanalysis would flag regions that were possibly missassembled.# primary subprograms
    find_klumps:                Find query klumps in a set of subject sequences
    scan_alignments:            Scan a SAM/BAM file for misassembled candidate regions

# visualization subprograms
    alignment_plot:             Plot the alignments at a specified region using a SAM/BAM file
    klump_plot:                 Create a figure illustrating the klumps on the subject sequences

# accessory subprograms
    combine_klumps:             Combine klump tsv files from multiple runs
    find_gaps:                  Create a tsv file containing the positions of gaps from a reference genome
    get_exons:                  Create a fasta file of exonic sequences from a list of genes
    kmerize:                    Find the coordinates of the query k-mers in the subject sequences
    klump_sizes:                Print out the expected klump sizes for each sequencePrimary Subprogramsfind_klumpsThefind_klumpssubprogram accepts a set of sequences from--queryand locates their substrings (i.e., k-mers) in the--subjectsequence(s).find_klumpsuses the locations of matching k-mers to constructklumps(i.e., clusters of matching k-mers). The two arguments that tend to have the largest effect on the results are--rangeand--min_kmers. The--rangeparameter sets the maximum number of base pairs two k-mers can be apart from one another while still being in the same klump. The--min_kmersis the minimum number of k-mers required to form a klump. As one may want to try different parameters when detecting klumps, aquery_map.gzis generated. This file can be directly fed tofind_klumpsusing the--query_mapparameter to avoid repeatedly searching the--subjectsequences for the same--querysequences. The--threadsparameter is used to parallelize the--querysearch in the--subjectsequences.--subject:                  Subject sequences in FAST[A|Q] format    
    --query:                    Query sequences in FAST[A|Q] format
    --query_map:                The query_map file generated from `find_klumps` or `kmerize`
    --output:                   Name of output file
    --range:                    Maximum distance two k-mers must be in order to be considered within the same klump (default = 1000)
    --min_kmers:                Minimum number of k-mers needed to form a klump (default = 1)
    --threads:                  Number of threads (default = 1)

    Advanced options:
        --ksize:                    Length of K-mer (must be an integer greater than 0) (default = 17)
        --query_count:              Minimum number of different queries a sequence is matched to (default = 1)
        --klump_count:              Minimum number of klumps a sequence must have (default = 1)
        --limit:                    Number of subject sequences to load into memory at once (default 10000)As guidance, you may want to runklump_sizes(see section onklump_sizesbelow) to have an idea on the expected number of k-mers a query sequence (i.e., a klump) is represented by.# using subject & query sequences
klumpy find_klumps --subject input1.fa.gz --query input2.fa --min_kmers 2 --range 50

# using the query_map.gz file directly
klumpy find_klumps --query_map query_map.gz --min_kmers 2 --range 50The output is a simple table in.tsvformat. The content of the file will look something below.# klumpy find_klumps --subject input1.fa.gz --query input2.fa --min_kmers 2 --range 50
Sequence	Seq_Length	Klump	Klump_Start	Klump_End	Query	Kmer_Count	Direction
seq1	    18072	    Klump1	5169	    5247	    query1	79	        F
seq2	    15037	    Klump1	979         1091	    query2	113	        RWhere the first line documents the command used to generate the file, followed by several columns. The columns are as followsSequence:       Name of the sequence
Seq_Length:     Length of the sequence
Klump:          Name of klump (e.g., klump1, klump2, etc..)
Klump_Start:    1-based start position of the klump in the sequence
Klump_End:      1-based end position of the klump in the sequence
Query:          Name of the query source for the klump
Kmer_Count:     Number of k-mers in the klump
Direction:      Orientation on the sequence the klump is found
Pair_Num:       1 or 2 depending if the sequence is from R1 or R2 data (only if data is paired-end, column is excluded otherwise)This table can then be used for both theklump_plotandalignment_plotsub-programs to visualize the klumps. The--outputargument is optional if there is a desired output name. If a--subjectargument is provided, it uses the--subjectinput name, and adds a_klumps.tsvextension to the name. Otherwise, the default output name isquery_klumps.tsv. As guidance for klump construction, the subprogramklump_sizes(see section onklump_sizesbelow) can be used to develop an idea on the expected klump size for each query sequence.Advanced options--query_countsets the minimum number of query sources a sequence must have to not be filtered out. For instance, if you are only interested in sequences that have klumps from >1 query sequence (such as the case when searching for sequences containing multiple exons), you may want to set--query_countto 2 or greater. Similarly, if you are only interested in sequences that have more than 1 klump, you could set--klump_countto 2 or greater. The--ksizeparameter sets the k-mer length during the k-mer mapping stage, and the--limitparameter regulates the number of sequences read into memory at once. By default,--limitis set to 10K sequences. However, if--limitis notexplicitlyused in the command,klumpywill lower the limit to 3 if it encounters sequences 1 Mb or longer.scan_alignmentsThescan_alignmentsprogram appliesklumpy's grouping algorithm across an assembly and aims to provide the user with locations that may be missassembled. By identifying which regions haveNor more groups (based on the criteria as described inalignment_plot), users are not restricted to having prior knowledge on which loci to investigate. The software takes the same grouping arguments asalignment_plot, with a couple additional parameters.--alignment_map:            A sorted and indexed SAM/BAM file [Required]
    --min_len:                  Minimum length a sequence must be to retain an alignment in the grouping analysis (default = 2000)
    --min_percent:              Minimum percent a sequence must be aligned for the alignment to be retained in the grouping analysis (default = 50)
    --num_of_groups:            Number of groups at a window to flag the window (default = 3)
    --threads:                  Number of threads (default = 1)
    --annotation:               Gene annotations in GTF/GFF3 format [Optional]

    Advanced options:
        --deletion_len:             Minimum length of deletion to draw and split alignment into blocks (default = 100)
        --assume_sep_del:           If set, remove the assumption that overlapping alignments with deletions can be automatically grouped if compatible
        --clip_tolerance:           Number of clipped base pairs to tolerate when comparing two alignments for compatibility (default = 400)
        --del_tolerance:            Length (in bp) covered by a deletion to be tolerated when comparing two alignments for compatibility (default = 50)
        --per_overlap:              Minimum percent two sequences need to overlap one another in order to confidently consider them compatible (default = 75)
        --t_len:                    Minimum length a sequence must be in order to trust its assignment when grouping (default = 2000)
        --t_per:                    Minimum percentage a sequence must be aligned in order to trust its assignment when grouping (default = 30)
        --align_offset:             Maximum number of base pairs apart for two sequences to be considered aligned at the same position (default = 100)
        --min_grp:                  Minimum number of sequences in a group in order to retain the group (default = 2)
        --window_size:              Length (in bp) for the sliding window to be (default = 50000)
        --window_step:              Length (in bp) for the sliding window to slide down to establish the next window (default = 25000) 
        --limit:                    Maximum number of alignments to process in a window (default 10000)With the exception of--threads, the main parameters are shared withalignment_plot. Here, a sliding window approach is implemented, and at each window,klumpy's grouping algorithm is performed (seeAdvanced optionsunder thealignment_plotsection). The use of multiple threads (i.e., the--threadsargument) is only useful if the alignment map contains more than 1referencesequence (parallelization is implemented across reference sequences, not alignment records).To perform a scan, one can run a command similar to thisklumpy scan_alignments --alignment_map input.bam --threads 16Here, we used the default parameters of using sequences with a length of at least 2 Kbp (set by--min_len), with the default--min_percentvalue of 50, meaning that sequences where more than a quarter of the sequence are unmapped, are not retained in the scanning process.The--num_of_groupsargument sets the minimum number of groups a region must have in order for the region to be flagged as a possible missassembly. If a.gtfor.gfffile is provided to--annotation, only regions containing a gene will be processed (useful if you are only interested in annotated regions).When running this software, temporary files will be written to disk using the naming schemeTemp_seq_name.txt, where theseq_nameis the name of the reference sequence being scanned.DO NOT DISGARD THESE FILES, they will be removed after the data is merged.The temporary files contain flaggedwindowsthat look something like thisseq 400000  425000
seq 750000  775000
seq 775000  800000Where the first column contains the sequence name, the second column the start position of the window, and the third column contains the end position of the window. If an annotation file is provided, the temporary file would contain an additional columnseq 400000  425000  geneA,geneB,geneC
seq 750000  775000  geneX,geneY
seq 775000  800000  geneY,geneZWhere the fourth column is added with a comma separated list containing the genes found in the window.Once all the reference sequences have been scanned, they are merged into a single file. The output file will contain the same name as in the alignment map, with the added extension of_Candidate_Regions.tsv. For example, if the input was calledinput.bam, the output would beinput_Candidate_Regions.tsv. Using the examples above, the resulting output would like this# klumpy scan_alignments --alignment_map input.bam --threads 16
Region_Num  Reference_Seq   Start   End
1   seq 400000  425000
2   seq 750000  800000

# with annotation file provided
# klumpy scan_alignments --alignment_map input.bam --threads 16 --annotation input.gtf.gz
Region_Num  Reference_Seq   Start   End Genes
1   seq 400000  425000  geneA,geneB,geneC
2   seq 750000  800000  geneX,geneY,geneZThe first line of the output will contain the line of code provided used to start the analysis, while the second line contains the column headers. Here,regionsare reported instead of windows sinceklumpywill collapse overlapping windows since they represent the same locus. ThReference_Seqcolumn contains the name of the reference sequence, and theStartandEndcolumns contain positions of the region (NOTEthese are 0-based positions). TheGenescolumn is optionally added if an annotation file was used in the analysis. If a desired window is of interest, the_Candidate_Regions.tsvoutput can be supplied toalignment_plotusing the--candidatesoption in combination with the--region_numargument to specify which regionklumpyshould plot instead of explicitly using--reference,--leftbound, and--rightbound.Advanced optionsThescan_alignmentssubprogram shares the majority of its arguments withalignment_plot, which both implement the grouping algorithm. New arguments include--window_sizeand--window_step. The software runs a sliding window across each reference sequence in the alignment map (i.e., aSAM/BAMfile), and applies the grouping algorithm at each window. The size of the window is determined by--window_size(defaults to 50 Kb) and the sliding distance is set by--window_step(default = 25 Kb). Reference sequences that are shorter than--window_sizeare ignored.Visualization Subprogramsklump_plotThis program is used to visualize the klumps from the.tsvfile generated fromfind_klumps. It only requires 1 argument, with a number of optional parameters to annotate the image or modify the output file.--klumps_tsv:               Klump tsv output generated from `find_klumps` [Required]
    --color:                    Color to use for drawing klumps (default = blue)
    --list_colors:              If set, the program will list the available colors for klumps and exit
    --fix_width:                If set, all the sequences in the figure will have the same `image` width
    --gap_file:                 File containing locations of gaps in the sequences (generated using `find_gaps`)
    --seq_name:                 Name of specific sequence to plot
    --format:                   Output format to draw image on. Choices are `pdf`, `png`, and `svg` (default = pdf)
    
    Advanced options:
        --klump_colors:             A tsv file containing the names of the query sources and the color to paint the query-specific klumps
        --color_by_gene:            If set and in combination with --klump_colors, will color all exons belonging to the same gene
        --leftbound:                Leftmost Position to plot --seq_name (default = 1) [requires --seq_name]
        --rightbound:               Rightmost Position to plot --seq_name (default = 50000) [requires --seq_name]
        --tick_count                If used, specify the number of ticks to use on the specified sequence [requires --seq_name]
        --tick_span                 If used, specify how many base pairs the ticks on the specified sequence should be apart [requires --seq_name]Using--list_colorswill list out a variety of colors to select from (currently at 64 options), to color the klumps (sequences are set to a light grey)klumpy klump_plot --list_colors # other arguments will be ignored if usedTo illustrateklump_plot's use, an example is shown belowklumpy klump_plot --klumps_tsv input_klumps.tsv --color blueThe resulting file is written by default in pdf format, but can be changed using the--formatoption, which accepts eitherpdf,pngorsvgas a value. The ouput file will have the same name as the file passed to--klumps_tsv. For example, the output forinput_klumps.tsvwould beinput_klumps.pdf. Below is an example of 6 sequencesSequences are sorted by sequence length, with the shortest sequence at the top, and the longest sequence at the bottom. Furthermore, since the number of sequences to plot is unknown, the height of the pdf is extended so that each sequence is equally spaced apart. The klump labels begin with the number of k-mers in the klump, separated by a-from the name of the query source.If using the--fix_widthflag is implemented, there will be no relative scaling done on the sequences.klumpy klump_plot --klumps_tsv input_klumps.tsv --color blue --fix_widthIn the case in which you are interested in a particular locus in a specific sequence, the--seq_nameflags can be used.klumpy klump_plot --klumps_tsv input_klumps.tsv --color blue --seq_name Seq4If the sequences contain gaps, the output offind_gapscan be used to annotate the figure with the sequence gaps (see./TestCases/TestCases.mdfor an example where we viewed gaps across severalBombusreference genomes)Advanced optionsThe--klump_colorsand--color_by_geneparameters accept a.tsvfile and uses that to map a color to a specific query source. If the color is unavailable, the program will report back to the user the error and exit. Not every query source needs to be present in the.tsvfile, as those unlisted sources will be defaulted to a blue color. An example of an acceptable tab-delimited file is shown belowQuery1  blue
Query2  maroon
Query3  teal
Query4  forestgreen
Query5  lavender
Query6  cyanAssuming the file is namedklump_colors.tsv, the following command illustrates its useklumpy klump_plot --klumps_tsv input_klumps.tsv --fix_width --klump_colors klump_colors.tsvThe file supplied to--color_by_geneis treated differently, as this argument was designed to work with the results of theget_exonssubprogram. Sequences obtained withget_exonswill have the extension_ENin the sequence header, whereEis short for Exon andNis for theNthexon. So if using the above example, all klumps withquery1_E[N]will be colored blue (Nbeing any number).To view a specific region in a specific sequence in the klumps.tsvfile, the--leftboundand--rightboundparameters can be used.klumpy klump_plot --klumps_tsv input_klumps.tsv --color blue --seq_name Seq4 --leftbound 5e3 --rightbound 7e3Further customization can be performed with the--tick_countand--tick_spanparameters, both of which are only implemented when viewing a single sequence.--tick_countspecifies the number of tick markers in the drawing, while--tick_spanwill set the length (in base pairs) between tick markers.klumpy klump_plot --klumps_tsv input_klumps.tsv --color blue --seq_name Seq4 --tick_count 20klumpy klump_plot --klumps_tsv input_klumps.tsv --color blue --seq_name Seq4 --tick_span 2e3kmerizeThekmerizesubprogram performs the--querysearch in the--subjectsequences implemented byfind_klumps. Aquery_map.gzfile will be generated, but no klump construction will be performed. All parameters available tokmerizeare those which are also available tofind_klumps(seefind_klumpssection for details ).--subject:                  Subject sequences in FAST[A|Q] format [Required]    
    --query:                    Query sequences in subject sequences, in FAST[A|Q] format [Required]
    --threads:                  Number of threads (default = 1)
    
    Advanced options:
        --ksize:                    Length of K-mer (must be an integer greater than 0) (default = 17)
        --limit:                    Number of subject sequences to load into memory at once (default 10000)The only required options are--subjectand--query. If there are multiple sequences in--subject, you can provide >1--threadsto perform a parallelized search. Here is an example of how to run the program.klumpy kmerize --subject input1.fa.gz --query input2.fa --threads 4The first line is a print out of the command used to run the analysis. The second line is used to keep track of the kmer size (which is used byfind_klumps). The third line specifies the time the analysis began, and the following line presents the format of the k-mer coordinates. Following, the lines start off with the name of the sequences in--subject(here,seq1andseq2), followed by the length of the sequence encapsulated in[]. After the colon, information on a k-mer found in the sequence that is shared with one of the sequences in--queryis listed, with the start position of that k-mer on the subject sequence, the orientation the k-mer is found in that sequence, and the name of the query sequence that the k-mer originated from.NOTE: k-mer positions are 0-based.So for subject sequenceseq1, the sequence has a length of 18,072 bp, and has 4 shared k-mers with query sequencequery1, each which are found on the reverse orientation (denotated asR). This component ofklumpyis computationally expensive, but once all the query k-mers are mapped onto the subject sequences, downstream analysis is quicker.If you are working with paired-end data, you will have to runkmerizeorfind_klumpson the two pairs of readsseparately, as trying to analyze them together delves into the field of sequence assemblies (e.g., contig creation), which there are a number of software that are designed for such tasks (e.g.,PEAR).klumpywill attempt to determine whether the data areR1reads orR2reads using the sequence header. More specifically, it assumes that sequences longer than 160 bp are not paired-end (this assumption may change in the future), and then looks for the following labels in the sequence header:/1,/2,[space]1:, or[space]2:. If present, the results are slightly changed to something like below# klumpy kmerize --subject input1.fa.gz --query input2.fa --threads 4
# k-size: 17
# Starting time: July 1, 2023: 12:10:45
# Sequence [Sequence Length]: Position_in_Seq Orientation_in_Seq Query_Source
seq1_PAIRED1 [18072]:  4968 R query1, 4969 R query1, 4970 R query1, 4971 R query1
seq2_PAIRED1 [15037]:  11837 F query2, 11838 F query2
# Ending time: July 1, 2023: 12:11:33Where_PAIRED1at the end of the sequence name would indicate that the sequence is the first of the pair (i.e., aR1sequence). A_PAIRED2extension would signal that the sequence is fromR2reads.Notes:klumpycurrently was not designed for protein sequences. Also, sequences that are shorter than--ksizeare ignored in the analysis.Advanced optionsBy default,--ksizeis set to 17 for all analysis inklumpy. Unless a specific--limitis set,klumpywill read 10,000--subjectsequences into memory at a time. If a sequence >=1 Mb in length is found and--limitis not explicitly used,klumpywill reduce the number of--subjectsequences processed at a single time to 3.klump_sizesThis piece of code only takes in two arguments--fasta:                    Sequences to break down to k-mers, in FASTA format [Required]
    --ksize:                    Length of k-mer (must be an integer greater than 0) (default = 17)The input is currently restricted to fasta files, but can be extended to fastq files if demand is high enough. As mentioned inkmerize, the default--ksizefor each sub-program is 17. To run the program, simply executeklumpy klump_sizes --fasta input.fa.gz --ksize 17No files are written and instead, the following lines are printedseq1 length 92: Klump size of 76 k-mers
seq2 length 223: Klump size of 207 k-mers
seq3 length 129: Klump size of 113 k-mers
seq4 length 129: Klump size of 113 k-mers
seq5 length 205: Klump size of 189 k-mersWhere the name of each sequence provided to--fastais printed next to its length, and the number of k-mers of--ksizethat the sequence can be broken down to. Sequences shorter than--ksizeare ignored. The main purpose of this program is to help guide your filteration infind_klumpswhen selecting a--min_kmersvalue.combine_klumpsIf you want to combine multiple klump.tsvfiles for, let's say,R1andR2reads, or raw reads and a reference genome, thencombine_klumpsshould be implemented. Only two arguments are accepted by this software--output:                   Name of output file
    --klumps_tsv_list:          Names of files to combine (--klumps_tsv_list file1.tsv file2.tsv file3.tsv etc...) [Required]The only required argument is--klumps_tsv_listwhich takes the files listed after the flag as its value. The--ouputargument is optional and by default, the resulting file is calledCombined_Klumps.tsv. An example of how to run the software with 3 files is shown belowklumpy combine_klumps --output Combined_Klumps.tsv --klumps_tsv_list input1_klumps.tsv input2_klumps.tsv input3_klumps.tsvThe resulting combined klumps file can then be used in theklump_plotandalignment_plotsubprograms.find_gapsThefind_gapssoftware that takes a fasta file (using--fasta), and generates atsvfile of the locations of gaps in the sequences. The motivation behind this tool is to map the gaps of a genome assembly in order to view this feature inalignment_plotorklump_plot. To runfind_gaps, simply runklumpy find_gaps --fasta reference_genome.fa.gzThe output will be named based on the input, with the added extension of_gaps.tsv. So in the example above, the output will be namedreference_genome_gaps.tsv. The output will look like the followingChrom   Start   End
ref_seq1 2758675 2759174
ref_seq1 3270081 3270580
ref_seq2 1400694 1401193
ref_seq2 2538532 2539031
ref_seq2 3770171 3770670
ref_seq3 996028 996527
ref_seq4 1299566 1300065
ref_seq4 2016232 2016731
ref_seq4 2871724 2872223
ref_seq5 5000507 5001006alignment_plotThis component ofklumpyhas the most arguments and generates one of the key outputs of the program. Here, the software is given a region to visualize, and creates an image of the alignments at the specified region. Examples will be illustrated using the raw Pac-Bio and RAD-Seq data and the assembled genome of the Mackerel Icefish from Rivera-Colón et al. 2023, which can be downloaded fromhere. Below are the currently available argumentsOptions for `alignment_plot`:
    --alignment_map:            A sorted and indexed SAM/BAM file [Required]
    --reference:                Name of reference sequence
    --leftbound:                Leftmost Position in the reference genome (default = 0)
    --rightbound:               Rightmost Position in the reference genome (default = 50000)
    --region_num:               Region Number in a *_Candidate_Regions.tsv file generated by `scan_alignments` (default = 1)
    --candidates:               A *_Candidate_Regions.tsv file generated from `scan_alignments`
    --min_len:                  Minimum length a sequence for retaining sequence in the plot (default = 2000)
    --min_percent:              Minimum percent a sequence must be aligned for it to be retained in the plot (default = 50)
    --klumps_tsv:               Klump tsv output generated from `find_klumps`
    --gap_file:                 File containing locations of gaps in the reference (generated using `find_gaps`)
    --color:                    Color to use for drawing klumps (default = blue)
    --annotation:               Gene annotations in GTF/GFF3 format
    --list_colors:              If set, the program will list the available colors for klumps and exit
    --vertical_line_gaps:       If set, a vertical line above gaps will be drawn
    --vertical_line_klumps:     If set, a vertical line above the reference klumps will be drawn
    --vertical_line_exons:      If set, a vertical line above the exons will be drawn
    --format:                   Output format to draw image on. Choices are `pdf`, `png`, and `svg` (default = pdf)
    --group_seqs:               If set, the sequences in the plot will be grouped based on alignment patterns

    Advanced options:
        --min_klump_size:           Minimum number of k-mers a klump must contain to retain in the plot (default = 1)    
        --max_percent:              Maximum percent a sequence must be aligned for it to be retained in the plot (default = 100) 
        --height:                   Height (in pixels) of the pdf to be generated (default = 2000)
        --width:                    Width (in pixels) of the pdf to be generated (default = 2000)
        --klump_colors:             A tsv file containing the names of the klump sources and the color to use for drawing these klumps
        --color_by_gene:            If set and in combination with --klump_colors, will color all klumps belonging to the same gene
        --plotting_list:            List of sequences to plot (filters remain on)
        --deletion_len:             Minimum length of deletion to draw and split alignment into blocks (default = 100)
        --view_span:                Length (in bp) to view from the leftbound and the rightbound when parsing SAM/BAM file (default = 1000000)
        --number:                   If set, sequences on the plot will be numbered 
        --tick_count                If used, specify the number of ticks to use on the reference genome [Optional]
        --tick_span                 If used, specify how many base pairs the ticks on the reference genome should be apart [Optional]
        --write_table:              If set, general information on the sequences on the plot will be written out to a TSV file
        --paired:                   Set if working with paired-end data
        --no_primary:               If set, primary alignments will be filtered out
        --no_subsample:             If set, will not randomly subsample alignments when the number of alignments is greater than --limit
        --supplementary:            If set, supplementary alignments will be kept if they pass all other filters
        --secondary:                If set, secondary alignments will be kept if they pass all other filters
        --write_groups:             If set, the sequences for each group will be written out in fasta format (one file per group)
        --write_seqs:               If set, the sequences that were plotted will be written in fasta format
        --write_edge_seqs:          If set, sequences with clipped alignments at the edges/ends of their groups will have their names written to a list
        --assume_sep_del:           If set, remove the assumption that overlapping alignments with deletions can be automatically grouped if compatible
        --clip_tolerance:           Number of clipped base pairs to tolerate when comparing two alignments for compatibility (default = 400)
        --del_tolerance:            Length (in bp) covered by a deletion to be tolerated when comparing two alignments for compatibility (default = 50)
        --limit:                    Maximum number of alignments to process in a group analysis and draw (default 10000)
        --per_overlap:              Minimum percent two sequences need to overlap one another in order to confidently consider them compatible
        --t_len:                    Minimum length a sequence must be in order to trust its assignment when grouping (default = 2000)
        --t_per:                    Minimum percentage a sequence must be aligned in order to trust its assignment when grouping (default = 30)
        --align_offset:             Maximum number of base pairs apart for two sequences to be considered aligned at the same position (default = 100)
        --min_grp:                  Minimum number of sequences in a group in order to retain the group (default = 10)The only required parameters are--alignment_map. The--alignment_mapaccepts a sorted and indexedSAMorBAMfile,which must be located in the same directory as its index file. Seeherefor details. The--leftboundand--rightboundvalues tellklumpywhich region in the--referenceto plot. Alternatively, one can supply the region number (--region_num) from a_Candidate_Regions.tsvfile (--candidates) generated fromscan_alignmentsto specify which locus to plot (seescan_alignmentsfor details).The--min_lenflag accepts an integer and sets the minimum length asequencemust be in order to be kept in the plot. Similarly,--min_percentsets the minimum percent asequencemust be aligned to the reference in order for it to be kept. The--klumps_tsvargument takes in the.tsvgenerated fromfind_klumpsorcombine_klumpsto annotate the sequences in the figure. If the.tsvfile contains klumps for both the aligned sequences and the reference sequence, klumps will be plotted onto both. A vertical bar can be plotted above klumps contained inreference sequenceby using the--vertical_line_klumpsflag. To see what colors are available for klumps, you can use--list_colorsto list the available colors (these are the same as the colors inklump_plot). The parameters used to control the color of the klumps are the same as those inklump_plot(see theklump_plotsection for details)No klumps are needed to runalignment_plot, and here we will demonstrate some of its usage on chromosome 24 in the Mackerel Icefish. Below, we plot the 4.47 Mb - 4.55 Mb region.klumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 24 \
                      --leftbound 4.47e6 \    # could use 4470000
                      --rightbound 4.55e6 \   # could use 4550000
                      --min_len 1e4 \         # can use 10000
                      --min_percent 50        # could use 0.50The output will resemble something along the lines of what is shown belowThe leftmost postitioned sequence is placed at the top, with the rightmost sequence positioned at the bottom (excluding the reference sequence). The alignments either are a light grey, indicating a forward alignment (in respect to the reference genome), or a shade of yellow, which represent alignments on the reverse strand of the reference genome. Several other features are shown here, notably deletions and clips.Zooming in, there are horizontal black lines that separate the blocks of the aligned portions of the sequence. These black lines represent deletions. Data such as Pac-Bio are known to have many small indels. As these may be due to sequencing errors as opposed to real biological indels, viewing these small indels may not benefit the user. By default, the--deletion_lenargument is set to 100 (Notethat this is an advanced parameter that). In other words,klumpywill draw deletions that are at least 100 bp in length. As insertions do not consume the reference sequence, they are ignored. The light green extensions from the alignment represent soft clips in the alignment, while red lines extending from the alignment would indicate hard clips (not shown in example).Additional annotation files that can be provided include--gap_file(file generated fromfind_gaps) and--annotation, which takes a.gtfor.gfffile (can be compresssed) as input. Both these two files will annotate the reference sequence with any features located within region. For further annotation,--vertical_line_gaps,--vertical_line_klumps, and--vertical_line_exonswill draw a dashed line above the desired feature in the reference sequence.Example with--annotation.klumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 24 \
                      --leftbound 4.47e6 \    # could use 4470000
                      --rightbound 4.55e6 \   # could use 4550000
                      --min_len 1e4 \         # could use 10000
                      --min_percent 50 \      # could use 0.50
                      --annotation cgun.gtf.gzThe output will resememble something along the lines of what is shown belowHere, each gene is differentiated by its color. For example, geneg_9529is red while geneg_16157is a lighter blue. Each separate annotation represents an exon, with the first exon in the image for each gene being tagged with the name of the gene. Gene labels alternate in position, where one gene's label is drawn on top of the reference sequence, and the following gene's label will be drawn below the reference sequence. If a third gene is drawn, its label will then be drawn above the reference sequence and the pattern continues.Plotting a region where a gap is located, we can see the followingklumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 24 \
                      --leftbound 13.07e6 \    # could use 13070000
                      --rightbound 13.15e6 \   # could use 13150000
                      --min_len 1e4 \         # could use 10000
                      --min_percent 50 \      # could use 0.50
                      --annotation cgun.gtf.gz \
                      --gap_file cgun_gaps.tsv \
                      --vertical_line_gaps    # no input neededHere, it is clear that the gap connecting two contigs is directly below a deletion in several alignments.To present an example with klumps, we will annotate one of theafgpregions in our assembly & alignments.klumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 3 \
                      --leftbound 3.7e6 \   # could use 3700000
                      --rightbound 4e6 \    # could use 4000000
                      --min_len 1.5e4 \     # could use 15000
                      --min_percent 75 \    # could use 0.75
                      --annotation cgun.gtf.gz \
                      --gap_file cgun_gaps.tsv \
                      --vertical_line_gaps \# no input needed
                      --klumps_tsv afgp_klumps.tsvHere, it is clear that there is a gapless region containing relatively few AFGP loci in the Mackerel IcefishThe--group_seqsargument does not accept any input (i.e., just include the parameter in your command) and groups the sequences usingklumpy's grouping algorithm (seeAdvanced optionsfor detials)Advanced optionsBy default, only primary alignments are visualized, with the available options--secondaryand--supplementarythat can relax this constraint. To exclude primary alignments, simply use the flag--no_primary.If you are interested in seeing poorly aligned sequences, the flag--max_percentwill cap the percentage a sequence can be aligned (e.g., if you are interested in seeing sequences with <= 50% of their bases aligned).If interested in plotting only a specific set of sequences,alignment_plotaccepts a list of sequence names from a file using--plotting_list, and excludes any sequences in the given region that is not in the list.NOTEadditional filters such as--min_lenand--min_percentare still applied/active.The list should be a single column list, with each sequence name occupying a single lineseq1_name
seq2_name
seq3_name
etc..The--view_spanargument is one that may not need much modification from its default value (1 Mb). Since clipped portions of an alignment are ignored when usingsamtools view,klumpycallssamtools view ... [leftbound - view_span]-[rightbound + view_span]to ensure that all alignments in the window (despite being clipped) are captured.The--heightand--widthflags can be used to manipulate the size of the image.NOTE: images are not guaranteed to display the alignment clearly. For example, if plotting regions with notably high coverage or low coverage, the pixel distripution may cause the drawings (both reference & alignments) to become distorted.klumpydoes try to work around these cases by adjusting the image size when encountering these cases, but this aspect ofklumpyremains an active line of development.If--numberis used, the aligned sequences will be numbered (starting from 0) from the leftmost alignment, to the rightmost alignment. This may be useful when trying to analyze specific alignments, where one can use the--write_tableflag to write a.tsvfile containing some basic information of each alignment, as shown below (NOTE: the below table is just an example). Additionally, if the sequences in the figure are of particular interest, they can be written out in fasta format by setting the--write_seqsflag (NOTE: only sequences from primary alignments are written).Sequence	Sequence_Length	Chrom	    Flag	Position	Percent_Aligned	Clipped_Start	Clipped_End	Seq_Num
seq1	    19655	        ref_seq	    0	    3704012	    100.0	        None	        None	    0
seq2	    17473	        ref_seq	    16	    3711306	    65.81	        S	            S	        1
seq3	    14617	        ref_seq	    16	    3706959	    98.5	        S	            S	        2
seq4	    14153	        ref_seq	    0	    3707105	    100.0	        None	        None	    3
seq5	    15096	        ref_seq	    0	    3709664	    100.0	        None	        None	    4
seq6	    15997	        ref_seq	    0	    3710641	    86.52	        S	            S	        5TheSequencecolumns contains the name of the aligned sequence, and theSequence_Lengthholds the length of the sequence. TheChromcolumns contains the name of the reference sequence. TheFLAGcolumn contains thesam flagof the alignment. ThePositioncolumn contains the leftmostalignedposition of the alignment, and thePercent_Alignedcolumn consists of the percentage the sequence was aligned. TheClipped_StartandClipped_Endcolumns can hold 3 different values:None,S, orH. If the sequence is not clipped (at the start or end of thealignment), then the value isNone. If there is a soft clip, the value is anS, withHindicating a hard clip. If using--number, aSeq_Numcolumn is added. If using--group_seqs, then aGroup_Numcolumn will be added (see below for details on--group_seqs). To generate a table like the one above, one just needs to set to flags. The output file will have the naming schemeResults_NumAlignments_records_Chrom_leftbound_rightbound.tsvwhereNumAlignmentsis the number of alignment records retained, andChrom, leftboundandrightboundindicating the location of the reference investigated.klumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 24 \
                      --leftbound 4.47e6 \    # could use 4470000
                      --rightbound 4.55e6 \   # could use 4550000
                      --min_len 1e4 \         # could use 10000
                      --min_percent 50 \      # could use 0.50
                      --annotation cgun.gtf.gz \
                      --number \              # no input needed
                      --write_table           # no input neededZoomed in for clarityTo plot paired-end data, youmustuse the--pairedflag. The plotting scheme is different for paired-end data, where the pair of sequences are plotted on the same line, andR1sequences are plotted in green, andR2are colored as blue.Illustrating an example with some Restriction-site Associated Sequencing (RAD-Seq) data.klumpy alignment_plot --alignment_map cgun_raw_reads_rad.bam \
                      --reference 24 \
                      --leftbound 1.69e5 \    # could use 169000
                      --rightbound 1.71e5 \   # could use 171000
                      --min_len 100 \         # min length should be set for short reads
                      --min_percent 50 \      # could use 0.50
                      --annotation cgun.gtf.gz \
                      --paired                # no input neededHere, the reads are aligned right at an exon ofcdk15, which is acyclin-dependent kinase.--group_seqsis used to signal toklumpyto attempt to group the alignments based on their alignment patterns. Before describing some of the parameters, it must be noted that the grouping algorithm isonlyapplied onprimaryalignments andnon-paired-enddata.The grouping algorithm first starts by trying to group "good" alignments (i.e., alignments without large deletions or clipping). This is based on the assumption that alignments in missassembled regions tend to have features such as clipping and deletions. Most of the grouping-specific featuers are designed to work with these features.First, the--assume_sep_delflag is used toturn offthe assumption that alignments with deletionsshouldbe grouped together if there is their deletions span the same region in the reference. The--per_overlaptakes a percentage (e.g., 50 or 0.50), which sets the minimum percentage that alignments need to overlap one another (i.e., both alignments need to overlap each other by--per_overlap) to beconsidereda good match (i.e., this alone does not result in the two alignments being grouped together).The--t_lenand--t_perarguments work similarly to--min_lenand--min_percent. The--t_lenargument is used to set the "trust" length, which anysequencebelow this length will be ignored when grouping as it is assumed to be untrustworthy. Likewise, anysequencewith an aligned percentage lower than--t_peris also ignored when grouping. The rationale for these parameters is to avoid trying to bin short and poorly aligned sequences that cannot be reliably placed into a single group.The--clip_toleranceargument is used to set the number of clipped base pairs that can be tolerated between two alignments. Below is a simple case.Here, a clip from alignment B is overlapping an aligned portion of alignment A. If--clip_toleranceis shorter than theclip overlap, the two alignments are considered incompatible. Otherwise, they can still be compatible.The--del_toleranceis similar to--clip_tolerance, except that it evaluates the deletion overlaps.If the value provided to--del_toleranceis shorter than thedeletion overlap, then the two sequences are considered incompatible.In--group_seqs, alignments that begin at the same position are assumed to represent the same locus, as it is reasonable to assume the two sequences are of the same origin if starting at the same site. Since two alignments may notexactlyalign at the same position, but can be loosely interpretted as aligning to the same position, the--align_offsetprovides some wiggle room.The offset applies both to the left and right side of the aligned portions of an alignment (here, alignment A). For visual purposes, only alignment A is being checked in the illustration, but this evaluation is also applied from the perspective of alignment B.To illustrate a simple case, we will apply the grouping algorithm to a 50 Kb region on chromosome 3 using default parametersklumpy alignment_plot --alignment_map cgun_raw_reads.bam \
                      --reference 3 \
                      --leftbound 10.2e6 \      # could use 4470000
                      --rightbound 10.25e6 \    # could use 4550000
                      --min_len 2.5e4 \         # 25 Kb
                      --min_percent 75 \        # same as 0.75
                      --annotation cgun.gtf.gz \
                      --group_seqs              # no input neededHere, the sequence alignments are colored based on the group they were assigned to. Alignments that are ungrouped are left white. Here, there are two groups.To illustrate an unresolved region, we will focus on a region where the algorithm struggles to find 2 or less groups while keeping the parameters the same. Additionally, we will annotate the image with the gaps in the region.klumpy alignment_plot --alignment_map cgun_raw_reads.bam \  # using a different chrom for example
                      --reference 3 \
                      --leftbound 6.9e6 \           # 6.9 Mb
                      --rightbound 7.08e6 \         # 7.08 Mb
                      --min_len 2.5e4 \             # 25 Kb
                      --min_percent 85 \            # same as 0.85
                      --annotation cgun.gtf.gz \
                      --group_seqs \                # no input needed
                      --gap_file cgun_gaps.tsv \
                      --vertical_line_gaps          # no input neededWithin this 180 Kb region, 4 contigs are stitched together as indicated by the 3 gaps (stretches ofN's). Interestingly, the third contig (~6.99 Mb - 7.65 Mb) was predicted to be composed of three groups, with two of the groups having some overlap with each other. In this example,klumpypredicts 12 groups within this region. Although this is likely to not be the true number of groups, the binning of sequences into their respective groups can aid in the reconstruction of problematic loci.The remaining grouping-based arguments are--min_grp,--write_groupsand--write_edge_seqs. The--min_grpflag accepts an integer to set as the minimum number of sequences required to form a group. The--write_groupsflag is set to tell the program to write the groups to group-specific fasta files, which can be used for tasks such as local reassemblies or annotations. If your interests are more on clipped sequences at the ends/edges of the groups, implementing the--write_edge_seqswill write out the names of the clipped sequences at the ends/edges of the groups into group-specific.txtfiles. The following image highlights some of the edge sequences from above.get_exonsThe purpose of this subprogram is to take a list of genes, and extract out their exonic sequences from a reference assembly. All arguments are required for this program.--fasta:                    Reference genome in fasta format [Required]
    --annotation:               Gene annotations in GTF/GFF3 format [Required]
    --genes:                    Names of genes from which to extract exons from [Required] (e.g., --genes gene_name1 gene_name2 gene_name3)Usage ofget_exonsis shown bellowklumpy get_exons --fasta reference.fa.gz --annotation ref_annotations.gtf.gz --genes geneA geneB geneCThe resulting output is a fasta file calledexon_sequences.fa. The name of the sequences follow the scheme>gene_name_EN, wheregene_nameis the name of the gene,Eis short for exon, andNis theNthexon of that gene (e.g., geneA_E5 would be short for exon 5 of gene A). In the case where the gene is duplicated (i.e., there are multiple genes with the same name in the annotation file), then the name would use the naming scheme>gene_name_M_ENwhereMis theMthcopy of that gene found in the annotation file. If the gene is on the reverse strand (denoted as-in the annotation file), then the extracted sequence is reverse complemented.AuthorsGiovanni MadrigalDepartment of Evolution, Ecology, and BehaviorUniversity of Illinois at Urbana-Champaigngm33@illinois.eduBushra Fazal MinhasInformatics ProgramUniversity of Illinois at Urbana-Champaignbfazal2@illinois.eduJulian CatchenDepartment of Evolution, Ecology, and BehaviorUniversity of Illinois at Urbana-Champaignjcatchen@illinois.edu