Package: graph-part
GraphPartBiological sequence dataset partitioning methodGraph-Part is a Python package for generating partitions (i.e. train-test splits, or splits for cross-validation) of biological sequence datasets. It ensures minimal homology between different partitions, while balancing partitions for labels or other desired criteria.Preprint:https://www.biorxiv.org/content/10.1101/2023.04.14.536886v1InstallationGraphPart relies onneedleallfrom theEMBOSSpackage for Needleman-Wunsch alignments of sequences. Please refer to the official EMBOSS documentation for installation methods.
Additionally, GraphPart supportsMMseqs2for alignments. To use other algorithms that compute pairwise similarity measures, please refer to theprecomputedmode.We recommend to install GraphPart in a conda environment, and install EMBOSS frombioconda. The same goes forMMseqs2.conda install -c bioconda emboss

# If you also want to use the MMseqs2 mode
conda install -c conda-forge -c bioconda mmseqs2Alternatively, on Ubuntu, EMBOSS is available directly viasudo apt-get install emboss.To install GraphPart, runpip install graph-partThe commandgraphpartwill now be available on your command line.Alternatively, you can install GraphPart from source:conda install -c bioconda emboss
git clone https://github.com/graph-part/graph-part.git
cd graph-part
pip install .InstructionsCommand line interface for FASTA dataAs an example, this is a basic command for partitioning a dataset at a maximum pairwise cross-partition identity of 30% into 5 folds. The resulting partitions are balanced for equal frequencies of the class labels specified inlabel=in theFASTA headers.--threadscan be adapted according to your system and has no effect on the partitioning itself, it only affects the resource usage and processing speed.graphpart needle --fasta-file netgpi_dataset.fasta --threshold 0.3 --out-file graphpart_assignments.csv --labels-name label --partitions 5 --threads 12Alternatively, a train-validation-test split of the data can be made instead of folds:graphpart needle --fasta-file netgpi_dataset.fasta --threshold 0.3 --out-file graphpart_assignments.csv --labels-name label --test-ratio 0.1 --val-ratio 0.05 --threads 12Python APIA tutorial notebook showcasing how to use GraphPart from within Python is included attutorial.ipynb. The tutorial also covers partitioning of small molecule data.Input formatGraphPart works on FASTA files with a custom header format, e.g.>P42098|label=CLASSA|priority=0
MAPSWRFFVCFLLWGGTELCSPQPVWQDEGQRLRPSKPPTVMVECQEAQLVVIVSKDLFGTGKLIRPADL
>P0CL66|label=CLASSB|priority=1
MKKYLLGIGLILALIACKQNVSSLDEKNSVSVDLPGEMKVLVSKEKNKDGKYDLIATVDKLELKGTSDKNAlternatively , ":" and " - " (note there are spaces on either side of the-) can be used as separators instead of "|". It should be taken care that the sequence identifiers themselves contain no separator symbols. The keywordslabelandprioritycan be customized by specifying the--labels-nameand--priority-namearguments. Both elements of the header are optional, GraphPart can also just partition based on sequences alone, without any class balancing.You can find a script to convert.csvdatasets into the custom.fastaformat atcsv_to_fasta.pyOutput formatGraphPart produces a.csvfile that contains the cluster assignment for each sequence. Columnclustercontains the partition number. Removed sequences are not contained in the output file.AC,priority,label-val,between_connectivity,cluster
P42098,False,0,0,0.0
Q6LEM5,False,0,0,0.0
Q9JI81,False,0,0,1.0APIMinimal command:graphpart ALIGNMENT_MODE -ff FASTAFILE.fasta -th 0.3Supported alignersAlignment modeDescriptionneedleUse EMBOSS needleall to compute exact pairwise global Needleman-Wunsch (NW) identities for all sequences.mmseqs2Use MMseqs2 to compute fast identities from local alignments. Use with caution for nucleotides, as there it cannot be guaranteed that MMseqs2 computes all pairwise alignments.precomputedUse a list of precomputed identities or other similarity/distance metrics.mmseqs2needleUses MMseqs2 to compute fast identities, followed by recomputation of NW identities of all identities between the separation threshold and an user-defined lower threshold.ArgumentsSharedLongShortDescription--fasta-file-ffPath to the input fasta file, formatted according tothe input format.--out-file-ofPath at which to save the partition assignments as.csv. Defaults tographpart_result.csv.--threshold-thThe desired partitioning threshold, should be within the bounds defined by the metric.--partitions-paNumber of partitions to generate. Defaults to 5.--transformation-tfTransformation to apply to the similarity/distance metric. GraphPart operates on distances, therefore similarity metrics need to be transformed. Can be any ofone-minus,inverse,square,log,None. See thesourcefor definitions. As an example, when operating with sequence identities ranging from 0 to 1, the transformationone-minusyields corresponding distances. Defaults toone-minus.--priority-name-pnThe name of the retention priority in the meta file. Is either=0or=1. If specified, the algorithm first tries to reach the treshold by removing/moving low-priority (0) samples before proceeding to1samples.--labels-name-lnThe name of the label in the meta file. Used for balancing partitions.--initialization-mode-imUse either slow or fast restricted nearest neighbor linkage or no initialization. Can be any ofslow-nn,fast-nn,simple. Defaults toslow-nn.--no-moving-nmBy default, the removing procedure tries to relocate sequences to another partition if it finds more within-threshold neighbours in any. This flag disallows moving. In high-redundancy datasets, moving can lead to imbalanced partitions and should be disabled.--save-checkpoint-path-scOptional path to save the computed identities above the chosen threshold as an edge list. Can be used to quickstart runs in theprecomputedmode. Defaults toNonewith no file saved.--test-ratio-teMake a train-val-test split instead of partitions for cross-validation. Overrides--partitionswhen specified. Defaults to 0. Needs to be a multiple of 0.05.--val-ratio-vaMake a train-val-test split instead of partitions for cross-validation. Overrides--partitionswhen specified. Defaults to 0. Needs to be a multiple of 0.05.needleLongShortDescription--denominator-dnDenominator to use for percent sequence identity computation. The number of perfect matching positions is divided by the result of this operation. Can be any ofshortest,longest,mean,full,no_gaps. The first three options are computed from the original lengths of the aligned sequences.fullrefers to the full length of the alignment, including gaps, and is the default.no_gapssubtracts gaps from the full alignment length.--threads-ntThe number of threads to run in parallel. If-1, will use all available resources. Defaults to 1.--chunks-ncThe number of chunks into which to split the fasta file for multithreaded alignment. Defaults to 10.--parallel-mode-pmThe Python parallelization strategy to use.multithreadormultiprocess. Multiprocessing is potentially faster (especially for short sequences), but increases memory usage. Defaults tomultithread--nucleotide-nuUse this flag if the input contains nucleotide sequences. By default, assumes proteins.--triangular-trOnly compute triangular of the full distance matrix. Twice as fast, but can yield slightly different results if an alignment has two different solutions with the same score, but different identities. In some cases, a pairwise identity can be slightly above the threshold in one solution, and slightly below in another (e.g A:B = 29.8%, B:A = 30.4%).--gapopen-gapopen[10.0 for any sequence] The gap open penalty is the score taken away when a gap is created. The best value depends on the choice of comparison matrix. The default value assumes you are using the EBLOSUM62 matrix. (Floating point number from 1.0 to 100.0)--gapextend-gapextend[0.5 for any sequence] The gap extension penalty is added to the standard gap penalty for each base or residue in the gap. This is how long gaps are penalized. Usually you will expect a few long gaps rather than many short gaps, so the gap extension penalty should be lower than the gap penalty. An exception is where one or both sequences are single reads with possible sequencing errors in which case you would expect many single base gaps. You can get this result by setting the gap open penalty to a very low value and using the gap extension penalty to control gap scoring. (Floating point number from 0.0 to 10.0)--endextend-endextend[0.5 for any sequence] The end gap extension, penalty is added to the end gap penalty for each base or residue in the end gap. This is how long end gaps are penalized. (Floating point number from 0.0 to 10.0)--endweight-endweightFlag. Apply end gap penalties.--endopen-endopen[10.0 for any sequence] The end gap open penalty is the score taken away when an end gap is created. The best value depends on the choice of comparison matrix. The default value assumes you are using the EBLOSUM62 matrix for protein sequences. (Floating point number from 1.0 to 100.0)--matrix-datafileThis is the scoring matrix file used when comparing sequences. By default it is the file 'EBLOSUM62'. These files are found in the 'data' directory of the EMBOSS installation.mmseqs2LongShortDescription--denominator-dnDenominator to use for percent sequence identity computation. The number of perfect matching positions is divided by the result of this operation. Can be any ofshortest,longest,n_aligned.n_alignedis the length of the alignment. Use this with caution, as GraphPart doesn't use coverage controls in the mmseqs2 mode. Defaults toshortest.--nucleotide-nuUse this flag if the input contains nucleotide sequences. By default, assumes proteins. Use with caution! Not guaranteed to compute all pairwise alignments.--prefilter-prUse MMseqs2 prefiltering at the highest sensitivity instead of forcing computation of all-vs-all alignments.precomputedLongShortDescription--edge-file-efPath to a comma separated file containing precomputed pairwise metrics, the first two columns should contain sequence identifiers specified in the--fasta-file. This is can be used to run GraphPart with an alignment tool different from the defaultneedleallandmmseqs.--metric-column-mcSpecifies in which column the metric is found. Indexing starts at 0, defaults to 2 when left unspecified.mmseqs2needleThis mode combinesneedleandmmseqs2. After a first run ofmmseqs2, all pairwise alignments with an indentity belowrecompute-thresholdare computed usingneedle. Only then the partitioning is performed. Note that ifrecompute-thresholdis very low, it can be faster to just run inneedlemode. All arguments for the two modes are reused here, with the following exceptions:LongShortDescription--recompute-threshold-reThe threshold for MMseqs2 above which alignments should be recomputed using needleall. Has to be a number lower than--threshold.--denominator-mmseqs-dnmReplaces--denominator. Applies to the mmseqs2 alignment step.--denominator-needle-dnnReplaces--denominator. Applies to the needle alignment step.CitationGraphPart: Homology partitioning for biological sequence analysis
Felix Teufel, Magnús Halldór Gíslason, José Juan Almagro Armenteros, Alexander Rosenberg Johansen, Ole Winther, Henrik Nielsen
bioRxiv 2023.04.14.536886; doi: https://doi.org/10.1101/2023.04.14.536886FAQHow should I pickchunks?chunksshould be picked so that allthreadsare utilized. Each chunk is aligned to each other chunk, sothreads<=chunks*chunksresults in full utilization.I want to test multiple thresholds and partitioning parameters - How can I do this efficiently ?When constructing the graph, we only retain identities that are larger than the selectedthreshold, as only those form relevant edges for partitioning the data. All other similarities are discarded as they are computed. To test multiple thresholds, the most efficient way is to first try the lowest threshold to be considered and save the edge list by specifying--save-checkpoint-path EDGELIST.csv. In the next run, usegraphpart precomputed -ef EDGELIST.csvto start directly from the previous alignment result.GraphPart starts with nicely balanced partitions, but after homology removal the sizes are very imbalanced.By default, GraphPart tries to retain as many sequences as possible. In cases where the initialization clustering is far away from a valid solution (this happens when there are a lot of classes, with potentially small counts, and when there is high overall sequence similarity in the data), moving sequences between partitions will cause some partitions to grow large at the expense of others. You can try--no-movingto prevent this behaviour.I want to use a different similarity metric than sequence identity.GraphPart can be used with any similarity or distance metric. To do so, you need to provide a list of precomputed pairwise similarities in theprecomputedmode. The first two columns of the file should contain the sequence identifiers specified in the--fasta-file. The third column should contain the similarity metric. The--metric-columnargument can be used to specify the column index. If you want to use a similarity metric, you need to specify a transformation using--transformation. See thesourcefor definitions. As an example, when operating with sequence identities ranging from 0 to 1, the transformationone-minusyields corresponding distances. If your metric is a distance, you can use--transformation Noneto skip the transformation step.I want to use a different alignment tool than EMBOSS needleall or MMseqs.This is supported by theprecomputedmode. Please refer to the answer above.How does the moving step decide to which partition to move a sequence to?After initialization of the partitions, GraphPart iteratively moves sequences between partitions and removes sequences from the data to achieve homology separation. For each sequence, we compute how many connections it has to sequences in each other partition. If there are partitions with more connections than the current partition, the sequence is moved to the partition with the maximum number of connections. If there is a tie in the number of connections, the sequence is moved to the partition that appeared first when iterating the underlying graph data structure. We do not explicitly control this order.