Package: token2index
:zap: :card_index: token2index: A lightweight but powerful library for token indexingtoken2indexis a small yet powerful library facilitating the fast and easy creation of a data structure mapping
tokens to indices, primarily aimed at applications for Natural Language Processing. The library is fully tested, and
does not require any additional requirements. The documentation can be foundhere, some feature highlights are
shown below.Who / what is this for?This class is written to be used for NLP applications where we want to assign an index to every word in a sequence e.g. to be later used to look up corresponding
word embeddings. Building an index and indexing batches of sequences for Deep Learning models using frameworks like PyTorch or Tensorflow are common steps but are often written from
scratch every time. This package provides a ready-made package combining many useful features, like reading vocabulary files, building indices from a corpus or indexing entire batches in one single
function call, all while being fully tested.:sparkles: Feature HighlightsBuilding and extending vocabOne way to build the index from a corpus is using the build() function:>>>fromt2iimportT2I>>>t2i=T2I.build(["colorless green ideas dream furiously","the horse raced past the barn fell"])>>>t2iT2I(Size:13,unk_token:<unk>,eos_token:<eos>,pad_token:<pad>,{'colorless':0,'green':1,'ideas':2,'dream':3,'furiously':4,'the':5,'horse':6,'raced':7,'past':8,'parn':9,'fell':10,'<unk>':11,'<eos>':12,'<pad>':13})The index can always be extended again later usingextend():>>>t2i=t2i.extend("completely new words")T2I(Size:16,unk_token:<unk>,eos_token:<eos>,pad_token:<pad>,{'colorless':0,'green':1,'ideas':2,'dream':3,'furiously':4,'the':5,'horse':6,'raced':7,'past':8,'barn':9,'fell':10,'completely':13,'new':14,'words':15,'<unk>':16,'<eos>':17,'<pad>':18})Both methods and index() also work with an already tokenized corpus in the form of[["colorless", "green", "ideas", "dream", "furiously"], ["the", "horse", "raced", "past", "the", "barn", "fell"]]Easy indexing (of batches)Index multiple sentences at once in a single function call!>>>t2i.index(["the green horse raced <eos>","ideas are a dream <eos>"])[[5,1,6,7,12],[2,11,11,3,12]]where unknown tokens are always mapped tounk_token.Easy conversion back to stringsReverting indices back to strings is equally as easy:>>>t2i.unindex([5,14,16,3,6])'the new <unk> dream horse'Automatic paddingYou are indexing multiple sentences of different length and want to add padding? No problem!index()has two
options available via thepad_toargument. The first is padding to the maximum length of all the sentences:>>>padded_sents=t2i.index(["the green horse raced <eos>","ideas <eos>"],pad_to="max")>>>padded_sents[[5,1,6,7,12],[2,12,13,13,13]]>>>t2i.unindex(padded_sents)[['the green horse raced <eos>','ideas <eos> <pad> <pad> <pad>']]Alternatively, you can also pad to a pre-defined length:>>>padded_sents=t2i.index(["the green horse <eos>","past ideas <eos>"],pad_to=5)>>>padded_sents[[5,1,6,12,13],[8,2,12,13,13]]>>>t2i.unindex(padded_sents)[['the green horse <eos> <pad>','past ideas <eos> <pad> <pad>']]Vocab from fileUsingT2I.from_file(), the index can be created directly by reading from an existing vocab file.
Refer to its documentationherefor more info.Fixed memory sizeAlthough thedefaultdictclass from Python'scollectionspackage also posses the functionality to map unknown
keys to a certain value, it grows in size for every new key.T2Imemory size stays fixed after the index is built.Support for special tokensTo enable flexibility in modern NLP applications,T2Iallows for an arbitrary number of special tokens (like a
masking or a padding token) during init!>>>t2i=T2I(special_tokens=["<mask>"])>>>t2iT2I(Size:3,unk_token:<unk>,eos_token:<eos>,pad_token:<pad>,{'<unk>':0,'<eos>':1,'<mask>':2,'<pad>':3})Explicitly supported programmer lazinessToo lazy to type? The library saves you a few keystrokes here and there. instead of callingt2i.index(...)you can
directly callt2i(...)to index one or multiple sequences. Furthermore, key functions likeindex(),unindex(),build()andextend()support strings or iterables of strings as arguments alike.:electric_plug: Compatibility with other frameworks (Numpy, PyTorch, Tensorflow)It is also ensured thatT2Iis easily compatible with frameworks like Numpy, PyTorch and
Tensorflow, without needing them as requirements:Numpy>>>importnumpyasnp>>>t=np.array(t2i.index(["the new words are ideas <eos>","the green horse <eos> <pad> <pad>"]))>>>tarray([[5,15,16,17,2,18],[5,1,6,18,19,19]])>>>t2i.unindex(t)['the new words <unk> ideas <eos>','the green horse <eos> <pad> <pad>']PyTorch>>>importtorch>>>t=torch.LongTensor(t2i.index(["the new words are ideas <eos>","the green horse <eos> <pad> <pad>"]))>>>ttensor([[5,15,16,17,2,18],[5,1,6,18,19,19]])>>>t2i.unindex(t)['the new words <unk> ideas <eos>','the green horse <eos> <pad> <pad>']Tensorflow>>>importtensorflowastf>>>t=tf.convert_to_tensor(t2i.index(["the new words are ideas <eos>","the green horse <eos> <pad> <pad>"]),dtype=tf.int32)>>>ttensor([[5,15,16,17,2,18],[5,1,6,18,19,19]])>>>t2i.unindex(t)['the new words <unk> ideas <eos>','the green horse <eos> <pad> <pad>']:inbox_tray: InstallationInstallation can simply be done usingpip:pip3 install token2index:mortar_board: CitingIf you usetoken2indexfor research purposes, please cite the library using the following citation info:@misc{ulmer2020token2index,
    title={token2index: A lightweight but powerful library for token indexing},
    author={Ulmer, Dennis},
    journal={https://github.com/Kaleidophon/token2index},
    year={2020}
}