Package: dna-spacy
AboutDNA spaCy is a machine learningspaCypipeline for processing DNA sequences found in FASTA files. By treating the classification of DNA as an NLP problem, we can leverage NLP libraries, such as spaCy to classify DNA sequences. This methodology has already been explored by otherscholars. To our knowledge, however, these methods have not been applied in a spaCy pipeline. We selected spaCy because of its wide use in industry and academia, speed, and its Python library is accessible and expandable.The initial layer of the DNA spaCy layer is a custom tokenizer designed to work with DNA sequences represented as long sequences of unseparated letters that correspond to nucleotides. Traditionally, scientists divide these sequences into kmers, or substrings of length k. This is important for downstream tasks like classification, genome assembly, metagenomics, and many other applications. Here, the kmers correspond to "words" in the DNA sequence. The KMER tokenizer allows the user to pass a custom kmer size to tokenize the text. In addition to this, the KMER Tokenizer also converts the kmer sequences into unique integers. The KMERs are then subdivided into windows of a pre-specified length, which help alleviate the problem of DNA sequences of different lengths. These stages are important because they are the way in which our encoding layer, a transformer, was trained.By default, the doc container represents the tokenized text (each KMER in sequential order). In order to access the original string input, you can usedoc._.untokenized.The windows are provided atdoc._.windows.The next layer in the pipeline is an embedding layer. This is handled by a transformer model that was trained on ~43 million DNA sequences from 23 marine wildlife species and a collection of bacteria reads. Each window is sent to the transformer model and encoded.A final step averages out the encodings of each window for a given input. This is stored indoc.vector.This library is meant to be used alongside DNA spaCy Project which allows for you to train downstream tasks, such as DNA classification. This can be performed on each individual window, rather than the doc as a whole, allowing for a nuanced classification of portions of a DNA sequence.Installationpipinstalldna-spacyUsageFirst, import theDNAclass and instantiate it. Here you can specify the kmer size and window size that you want to use. Currently, the transformer models only supported arekmer_size=7andwindow_size=10fromdna_spacyimportdna_spacyasdsnlp=ds.DNA(kmer_size=7,window_size=10)Next, pass a sequence to the class and print it off.dna="TTCGGGGTGGCCCTGCAGGCGACGGGCGCGCCTGCGGTCCTCCGTGAACCCGGTGCGGTGACGATGCGCGTGCACGACGTCGACCTGCGGGTCTCGGCCGGGGCGTTCTTCCAGGCCGGCCCGGCCGCGGCTGCGGCGCTGGTCGACCTC"doc=nlp(dna)print(doc)TTCGGGGTCGGGGTCGGGGTGGGGGTGGGGGTGGCGGTGGCCGTGGCCCTGGCCCTGGCCCTGGCCCTGCCCCTGCACCTGCAGCTGCAGGTGCAGGCGCAGGCGCAGGCGAAGGCGACGGCGACGGCGACGGCGACGGGGACGGGCACGGGCGCGGGCGCGGGCGCGGGCGCGCGCGCGCCCGCGCCTGCGCCTGCGCCTGCGCCTGCGCCTGCGGCTGCGGTTGCGGTCGCGGTCCCGGTCCTGGTCCTCGTCCTCCTCCTCCGCCTCCGTCTCCGTGTCCGTGACCGTGAACGTGAACGTGAACCTGAACCCGAACCCGAACCCGGACCCGGTCCCGGTGCCGGTGCCGGTGCGGGTGCGGGTGCGGTTGCGGTGGCGGTGACGGTGACGGTGACGGTGACGATGACGATGACGATGACGATGCCGATGCGGATGCGCATGCGCGTGCGCGTGCGCGTGCGCGTGCGCGTGCACGTGCACGTGCACGTGCACGAGCACGACCACGACGACGACGTCGACGTCGACGTCGACGTCGACGTCGACGTCGACCTCGACCTCGACCTGGACCTGCACCTGCGCCTGCGGCTGCGGGTGCGGGTGCGGGTCCGGGTCTGGGTCTCGGTCTCGGTCTCGGTCTCGGCCTCGGCCTCGGCCGCGGCCGGGGCCGGGGCCGGGGCCGGGGCCGGGGCGGGGGCGTGGGCGTTGGCGTTCGCGTTCTCGTTCTTGTTCTTCTTCTTCCTCTTCCACTTCCAGTTCCAGGTCCAGGCCCAGGCCCAGGCCGAGGCCGGGGCCGGCGCCGGCCCCGGCCCCGGCCCGGGCCCGGGCCCGGCCCCGGCCCCGGCCGCGGCCGCGGCCGCGGCCGCGGCCGCGGCCGCGGCTGCGGCTGCGGCTGCGGCTGCGGCTGCGGCTGCGGCTGCGGCGGCGGCGCCGGCGCTGGCGCTGGCGCTGGCGCTGGTGCTGGTCCTGGTCGTGGTCGAGGTCGACGTCGACCTCGACCTCGACCTCYou can also access the original untokenized text via the custom extensiondoc._.untokenizedprint(doc._.untokenized)TTCGGGGTGGCCCTGCAGGCGACGGGCGCGCCTGCGGTCCTCCGTGAACCCGGTGCGGTGACGATGCGCGTGCACGACGTCGACCTGCGGGTCTCGGCCGGGGCGTTCTTCCAGGCCGGCCCGGCCGCGGCTGCGGCGCTGGTCGACCTCTheDoccontainer also contains custom extensions that allow you to access the individual windows:forwindowindoc._.windows:print(window)TTCGGGGTCGGGGTCGGGGTGGGGGTGGGGGTGGCGGTGGCCGTGGCCCTGGCCCTGGCCCTGGCCCTGCCCCTGCACCTGCAGCTGCAGGTGCAGGCGCAGGCGCAGGCGAAGGCGACGGCGACGGCGACGGCGACGGG...Each window has its own vectorforwindow_vecindoc._.window_vectors:print(window_vec)[[-7.39537120e-016.82567656e-012.06507981e-01-2.69144595e-01...,-7.39537120e-016.82567656e-012.06507981e-01-2.69144595e-01...]]The library also averages out each window to capture the entire sequence's vector. It automatically places this in thedoc.vectorextension.print(doc.vector)[[-7.39537120e-016.82567656e-012.06507981e-01-2.69144595e-01-5.84327400e-01-4.37274933e-01-9.49760228e-026.02423847e-01-2.15590253e-01-1.82508811e-01-2.84874998e-02-1.85540672e-02-3.08907479e-013.78764182e-01-3.48590463e-011.51973039e-01...]]