Saravanan666/pypi-org huggingface dataset:
Package: tokenize-output
Get identifiers, names, paths, URLs and words from the command output.Thexontrib-output-searchforxonsh shellis using this library.If you like the idea click ⭐ on the repo and stay tuned by watching releases.Installpipinstall-Utokenize-outputUsageYou can usetokenize-outputcommand as well as export the tokenizers in Python:fromtokenize_output.tokenize_outputimport*tokenizer_split("Hello world!")# {'final': set(), 'new': {'Hello', 'world!'}}Words tokenizingecho"Try https://github.com/xxh/xxh"|tokenize-output-p# Try# https://github.com/xxh/xxhJSON, Python dict and JavaScript object tokenizingecho'{"Try": "xonsh shell"}'|tokenize-output-p# Try# shell# xonsh# xonsh shellenv tokenizingecho'PATH=/one/two:/three/four'|tokenize-output-p# /one/two# /one/two:/three/four# /three/four# PATHDevelopmentTokenizersTokenizer is a functions which extract tokens from the text.PriorityTokenizerText exampleTokens1dict{"key": "val as str"}key,val as str2envPATH=/bin:/etcPATH,/bin:/etc,/bin,/etc3splitSplit  me \n now!Split,me,now!4strip{Hello}!.HelloYou can create your tokenizer and add it totokenizers_allintokenize_output.py.Tokenizing is a recursive process where every tokenizer returnsfinalandnewtokens.
Thefinaltokens directly go to the result list of tokens. Thenewtokens go to all
tokenizers again to find new tokens. As result if there is a mix of json and env data
in the output it will be found and tokenized in appropriate way.How to add tokenizerYou can start fromenvtokenizer:Prepare regexpPrepare tokenizer functionAdd the function to the listandto the preset.Add test.Now you can test and debug (see below).Test and debugRun tests:cd~
gitclonehttps://github.com/anki-code/tokenize-outputcdtokenize-output
python-mpytesttests/To debug the tokenizer:echo"Hello world"|./tokenize-output-pRelated projectsxontrib-output-searchforxonsh shell
Package: tokenizer-adapter
Tokenizer AdapterA simple tool for adapting a pre-trained Huggingface model to a new vocabulary with (almost) no training.This technique can significantly reduce sequence length when a language model is used on data with a specific vocabulary (biology, medicine, law, other languages, etc...).Should work for most Huggingface Hub language models (requires further testing).Everything is run on CPU.Installpip install tokenizer-adapter --upgradeUsageIt is recommended to use an existing tokenizer to train the new vocabulary.Best and easiest way is to use thetokenizer.train_new_from_iterator(...)method.fromtokenizer_adapterimportTokenizerAdapterfromtransformersimportAutoTokenizer,AutoModelForMaskedLMBASE_MODEL_PATH="camembert-base"# A simple corpuscorpus=["A first sentence","A second sentence","blablabla"]# Load model and tokenizermodel=AutoModelForMaskedLM.from_pretrained(BASE_MODEL_PATH)tokenizer=AutoTokenizer.from_pretrained(BASE_MODEL_PATH)# Train new vocabulary from the old tokenizernew_tokenizer=tokenizer.train_new_from_iterator(corpus,vocab_size=300)# Default params should work in most casesadapter=TokenizerAdapter()# Patch the model with the new tokenizermodel=adapter.adapt_from_pretrained(new_tokenizer,model,tokenizer)# Save the model and the new tokenizermodel.save_pretrained("my_new_model/")new_tokenizer.save_pretrained("my_new_model/")To rely on a custom tokenizer (experimental), you may need to use thecustom_preprocessingargument.Example using a RoBERTa (similar to Phi-2) style tokenizer for a CamemBERT model:fromtokenizer_adapterimportTokenizerAdapterfromtransformersimportAutoTokenizer,AutoModelForMaskedLMBASE_MODEL_PATH="camembert-base"NEW_CUSTOM_TOKENIZER="roberta-base"# A simple corpuscorpus=["A first sentence","A second sentence","blablabla"]# Load model and tokenizermodel=AutoModelForMaskedLM.from_pretrained(BASE_MODEL_PATH)tokenizer=AutoTokenizer.from_pretrained(BASE_MODEL_PATH)# Also load this custom tokenizer to train the new onenew_tokenizer=AutoTokenizer.from_pretrained(NEW_CUSTOM_TOKENIZER)new_tokenizer=new_tokenizer.train_new_from_iterator(corpus,vocab_size=300)# CamemBERT tokenizer relies on '▁' while the RoBERTa one relies on 'Ġ'adapter=TokenizerAdapter(custom_preprocessing=lambdax:x.replace('Ġ','▁'))# Patch the model with the new tokenizermodel=adapter.adapt_from_pretrained(new_tokenizer,model,tokenizer)# Save the model and the new tokenizermodel.save_pretrained("my_new_model/")new_tokenizer.save_pretrained("my_new_model/")
Package: tokenizer-hub
# tokenizer-hubyoctol 乂卍oO煞氣ㄟtokenizerOo卍乂Tokenizers have the same interface of Jieba:`python from tokenizer_hub import XXX_tokenizer tokenizer = XXX_tokenizer()tokenizer.lcut('我来到北京清华大学')> ['我', '来到', '北京', '清华大学'] `
Package: tokenizers-collection
中文分词器集合一些中文分词器的简单封装和集合Free software: MIT licenseDocumentation:https://chinese-tokenzier-iterator.readthedocs.io.FeaturesTODO使用fromtokenizers_collection.configimporttokenizer_registryforname,tokenizerintokenizer_registry:print("Tokenizer:{}".format(name))tokenizer('input_file.txt','output_file.txt')安装pipinstalltokenizers_collection更新许可文件与下载模型因为其中有些模型需要更新许可文件（比如：pynlpir）或者需要下载模型文件（比如：pyltp），因此安装后需要执行特定的命令完成操作，这里已经将所有的操作封装成了一个函数，只需要执行类似如下的指令即可python-mtokenizers_collection.helperCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2018-08-28)First release on PyPI.
Package: tokenize-rt
tokenize-rtThe stdlibtokenizemodule does not properly roundtrip.  This wrapper
around the stdlib provides two additional tokensESCAPED_NLandUNIMPORTANT_WS, and aTokendata type.  Usesrc_to_tokensandtokens_to_srcto roundtrip.This library is useful if you're writing a refactoring tool based on the
python tokenization.Installationpipinstalltokenize-rtUsagedatastructurestokenize_rt.Offset(line=None, utf8_byte_offset=None)A token offset, useful as a key when cross referencing theastand the
tokenized source.tokenize_rt.Token(name, src, line=None, utf8_byte_offset=None)Construct a tokenname: one of the token names listed intoken.tok_nameorESCAPED_NLorUNIMPORTANT_WSsrc: token's source as textline: the line number that this token appears on.utf8_byte_offset: the utf8 byte offset that this token appears on in the
line.tokenize_rt.Token.offsetRetrieves anOffsetfor this token.converting to and fromTokenrepresentationstokenize_rt.src_to_tokens(text: str) -> List[Token]tokenize_rt.tokens_to_src(Iterable[Token]) -> stradditional tokens added bytokenize-rttokenize_rt.ESCAPED_NLtokenize_rt.UNIMPORTANT_WShelperstokenize_rt.NON_CODING_TOKENSAfrozensetcontaining tokens which may appear between others while not
affecting control flow or code:COMMENTESCAPED_NLNLUNIMPORTANT_WStokenize_rt.parse_string_literal(text: str) -> Tuple[str, str]parse a string literal into its prefix and string content>>>parse_string_literal('f"foo"')('f', '"foo"')tokenize_rt.reversed_enumerate(Sequence[Token]) -> Iterator[Tuple[int, Token]]yields(index, token)pairs.  Useful for rewriting source.tokenize_rt.rfind_string_parts(Sequence[Token], i) -> Tuple[int, ...]find the indices of the string parts of a (joined) string literalishould start at the end of the string literalreturns()(an empty tuple) for things which are not string literals>>>tokens=src_to_tokens('"foo" "bar".capitalize()')>>>rfind_string_parts(tokens,2)(0, 2)>>>tokens=src_to_tokens('("foo" "bar").capitalize()')>>>rfind_string_parts(tokens,4)(1, 3)Differences fromtokenizetokenize-rtaddsESCAPED_NLfor a backslash-escaped newline "token"tokenize-rtaddsUNIMPORTANT_WSfor whitespace (discarded intokenize)tokenize-rtnormalizes string prefixes, even if they are not parsed -- for
instance, this means you'll seeToken('STRING', "f'foo'", ...)even in
python 2.tokenize-rtnormalizes python 2 long literals (4l/4L) and octal
literals (0755) in python 3 (for easier rewriting of python 2 code while
running python 3).Sample usagehttps://github.com/asottile/add-trailing-commahttps://github.com/asottile/future-annotationshttps://github.com/asottile/future-fstringshttps://github.com/asottile/pyupgradehttps://github.com/asottile/yesqa
Package: tokenizertools
tokenizertoolsText file tokenizers that support multiple start states
and lexical tracking implemented as standard Python iterators.Classes provided:Tokenizer – Base class.RegexTokenizer – Python re-based tokenizer.TokenizeAhead – A look-ahead iterator that can wrap any tokenizer.OverviewThe classRegexTokenizerimplements a tokenizer using theremodule to recognize tokens in the input stream.
Tokens and actions are defined by rules.
The tokenizer calls user action functions associated with each rule.
In most cases, the user action function can simply be a@classmethodconstructor of a user-provided token class.Each rule is specified as a tuple.
The first element of the tuple is a regular expression that will
be compiled byreand used to match a token.
The second element of the tuple is a user-provided callable that
will be passed
the recognized text, along with the current lexical position.In this example, the user classTokenimplements constructors
as@classmethodfunctions, and these serve as the callables in
each lexical rule.Rules are specified in the class variablespec, which is a
list of rules.import tokenizertools as tt
class MyTokenizer(tt.RegexTokenizer):
    spec = [
        (r'[a-zA-Z][a-zA-Z0-9_]*',Token.type_ident), # idents and keywords
        (r'[0-9]+\.[0-9]+',Token.type_float), # floats
        (r'[0-9]+', Token.type_int), # ints
        (r'\s*',None), # ignore white space
    ]Nothing else needs to be defined.  All methods are inherited.
Instantiate a lexer and commence parsing.
The specification rules are compiled and cached on creation
of the first instance.:tokenizer = MyTokenizer()
with open('foo.bar') as f:
    token_stream = Lookahead(tokenizer.lex(f, f.name))
    compiled_stuff = my_parser.parse(token_stream)tokenizertools newsRelease history.15-Sept-20161.0 Time to put some mileage on it.Clean ups, PEP8-ification.
Improved exception handling for badbegin()state.16-June-20140.1a Initial alpha.
Package: tokenizer-tools
Tokenizer ToolsTools/Utils for NLP (including dataset reading, tagset encoding & decoding, metrics computing) | NLP 工具集（包含数据集读取、tagset 编码和解码、指标的计算等）Free software: MIT licenseDocumentation:https://tokenizer-tools.readthedocs.io.Features常见数据集格式的读取多种 Tagset 的编码和解码 [BMES 体系,BILUO 体系,IOB 体系]指标的计算功能语料集读写本软件提供了一种语料存储的磁盘文件格式（暂定名为 conllx）和内存对象格式（暂定名为 offset）。语料集读取任务：读取 corpus.collx 文件，遍历打印每一条语料。代码：fromtokenizer_tools.tagset.offset.corpusimportCorpuscorpus=Corpus.read_from_file("corpus.conllx")fordocumentincorpus:print(document)# document 就是单条语料对象语料集写入任务：将多条语料写入 corpus.conllx 文件代码：fromtokenizer_tools.tagset.offset.corpusimportCorpuscorpus_list=[corpus_item_one,corpus_item_two]corpus=Corpus(corpus_list)corpus.write_to_file("corpus.conllx")Document 属性和方法每一个单条语料都是一个 Document 对象，现在介绍这个对象所拥有的属性和方法属性text类型是 list， 代表文本的字段domain类型是 string， 代表领域function类型是 string， 代表功能点sub_function类型是 string，代表子功能点intent类型是 string， 代表意图entities类型是 SpanSet， 代表实体，下文有详细介绍方法compare_entities比较文本和实体是否匹配convert_to_md将文本和实体转换成 markdown 格式，用于文本化渲染输出SpanSet 属性和方法方法__iter__可以像列表一样访问，得到的每一个元素都是 Span 对象check_overlap检查 span 是否重叠Span 属性和方法属性startint, 从 0 开始，包含该位置endint， 从0开始，不包含该位置entitystring， 实体类型valuestring， 实体的值TODO改变项目的名字，tokenizer_tools 已经无法正确描述现在项目的功能CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2018-09-05)First release on PyPI.
Package: tokenizer-viz
Tokenizer VizTokenizer Viz is a Python package that generates HTML to visualize the tokenization of text. It highlights tokens with different colors and customizable styles, making it easier to understand how a text is tokenized.Project Layouttokenizer-viz/
│
├── tokenizer_viz/
│   ├── __init__.py
│   └── visualization.py
│
├── .gitignore
├── LICENSE
├── README.md
└── setup.py

<br>

## Installation

You can install the **`tokenizer-viz`** package using pip:

```bash
pip install tokenizer-vizUsageHere's a quick example of how to use the package:Usage with a provided encoder and decoderfromtokenizer_vizimportTokenVisualizationfromIPython.displayimportHTML# Define sample encoder and decoder functions for demonstration purposesdefsample_encoder(text):returnlist(text)defsample_decoder(token):returntoken# Initialize the TokenVisualization class with the encoder and decoder functionstoken_viz=TokenVisualization(encoder=sample_encoder,decoder=sample_decoder)# Define a sample text to visualize tokenization boundariessample_text="This is a sample text.\nIt has multiple lines."# Visualize the tokenization boundarieshtml=token_viz.visualize(sample_text)HTML(html)EXAMPLE OUTPUTTBD EXAMPLE IMAGEARGUMENTSTheTokenVisualizationclass accepts several optional
parameters to customize the appearance and layout of the tokens:cmap(defualt='Pastel2'),font_family(defualt='Courier New'),transparency(default=0.675),font_size(defualt='1.1em'),unk_token(defualt='???'),font_weight(defualt=300),padding(defualt='0px'),margin_right(defualt='0px'),border_radius(defualt='0px'),background_color(defualt='#F0F0F0'),Please refer to the class docstrings and method docstrings for a detailed description of each parameter.LicenseThis project is licensed under the MIT License.
Package: tokenizesentences
tokenizesentencesPython3 module to tokenize english sentences.Based on the answer of D Greenberg in StackOverflow:https://stackoverflow.com/questions/4576077/python-split-text-on-sentencesInstallationInstall with pippip3 install -U tokenizesentencesUsageIn [1]: import tokenizesentences

In [2]: m = tokenizesentences.SplitIntoSentences()

In [3]: m.split_into_sentences(
    "Mr. John Johnson Jr. was born in the U.S.A but earned his Ph.D. in Israel before joining Nike Inc. as an engineer. He also worked at craigslist.org as a business analyst."
    )

Out[3]: 
[
    'Mr. John Johnson Jr. was born in the U.S.A but earned his Ph.D. in Israel before joining Nike Inc. as an engineer.',
    'He also worked at craigslist.org as a business analyst.'
]
Package: tokenize_uk
Tokenize UKSimple python lib to tokenize texts into sentences and sentences to words. Small, fast and robust. Comes with ukrainian flavourFree software: MIT licenseDocumentation:https://tokenize_uk.readthedocs.org.FeaturesTokenize given text into sentencesTokenize given sentence into wordsWorks well with accented characters (like stresses) and apostrophesSuitable also for other languagesHistory0.1.0 (2016-05-29)First release on PyPI.
Package: tokenlearner-pytorch
tokenlearner-pytorchUnofficial PyTorch implementation ofTokenLearnerby Ryoo et al. from Google AI (abs,pdf)InstallationYou can install TokenLearner viapip:pip install tokenlearner-pytorchUsageYou can access theTokenLearnerclass from thetokenlearner_pytorchpackage. You can use this layer with a Vision Transformer, MLPMixer, or Video Vision Transformer as done in the paper.importtorchfromtokenlearner_pytorchimportTokenLearnertklr=TokenLearner(S=8)x=torch.rand(512,32,32,3)y=tklr(x)# [512, 8, 3]You can also useTokenLearnerandTokenFusertogether with Multi-head Self-Attention as done in the paper:importtorchimporttorch.nnasnnfromtokenlearner_pytorchimportTokenLearner,TokenFusermhsa=nn.MultiheadAttention(3,1)tklr=TokenLearner(S=8)tkfr=TokenFuser(H=32,W=32,C=3,S=8)x=torch.rand(512,32,32,3)# a batch of imagesy=tklr(x)y=y.view(8,512,3)y,_=mhsa(y,y,y)# ignore attn weightsy=y.view(512,8,3)out=tkfr(y,x)# [512, 32, 23, 3]TODOAdd support for temporal dimensionTImplementTokenFuserwithViTImplementTokenFuserwithViViTContributionsIf I've made any errors or you have any suggestions, feel free to raise an Issue or PR. All contributions welcome!!LicenseMIT
Package: tokenlib
This is generic support library for doing token-based authentication.  You
might use it to build a login system using bearer tokens, two-legged oauth, or
MAC Access authentication.Given a server-side master secret, you can serialize a dict of data into
an opaque, unforgeable authentication token:>>> token = tokenlib.make_token({"userid": 42}, secret="I_LIKE_UNICORNS")
>>> print token
eyJzYWx0IjogImY0NTU5NCIsICJleHBpcmVzIjogMTMyOTg3NTI2Ny4xNDQ5MzUsICJ1c2VyaWQiOiA0Mn0miXCe4NQQtXTE8NXSGcsL6dzSuQ==Later, you can use the same secret to verify the token and extract the
embedded data:>>> data = tokenlib.parse_token(token, secret="I_LIKE_UNICORNS")
>>> print data
{u'userid': 42, u'expires': 1329875384.073159, u'salt': u'1c033f'}Notice that the data includes an expiry time.  If you try to parse an expired
token, it will fail:>>> # Use now=XXX to simulate a time in the future.
>>> tokenlib.parse_token(token, secret="I_LIKE_UNICORNS", now=9999999999)
Traceback (most recent call last):
...
ValueError: token has expiredLikewise, it will fail if the token was constructed with a non-matching secret
key:>>> tokenlib.parse_token(token, secret="I_HATE_UNICORNS")
Traceback (most recent call last):
...
ValueError: token has invalid signatureEach token also has an associated “token secret”.  This is a secret key that
can be shared with the consumer of the token to enable authentication schemes
such as MAC Access Authentication of Two-Legged OAuth:>>> key = tokenlib.get_token_secret(token, secret="I_LIKE_UNICORNS")
>>> print key
EZslG8yEYTGyDvBjRnxGipL5Kd8=For applications that are using the same settings over and over again, you
will probably want to create a TokenManager object rather than using the
module-level convenience functions:>>> manager = tokenlib.TokenManager(secret="I_LIKE_UNICORNS")
>>> data = manager.parse_token(token)
>>> print data
{u'userid': 42, u'expires': 1329875384.073159, u'salt': u'1c033f'}This will let you customize e.g. the token expiry timeout or hash module
without repeating the settings in each call.2.0.0 - 2017-12-20Py27, Py35 compliant0.3.1 - 2014-01-08Make tokenlib.DEFAULT_HASHMOD a string.Fix sporadically-failing test.0.3.0 - 2014-01-08clarified licensing infoswitched to sha256 as default hashing algorithmrenamedget_token_secrettoget_derived_secretto clarify its
purpose; the previous name remains but is deprecated.added custom ValueError subclasses for more fine-grained error reporting.0.2.0 - 2012-11-27support for python3 via source-level compatibility0.1.0 - 2012-03-14Initial release.
Package: tokenlists
py-tokenlistsUniswap Token Listsimplementation in Python.Dependenciespython3version 3.8 or greater, python3-devInstallationviapipYou can install the latest release viapip:pipinstalltokenlistsviasetuptoolsYou can clone the repository and usesetuptoolsfor the most up-to-date version:gitclonehttps://github.com/ApeWorX/py-tokenlists.gitcdpy-tokenlists
python3setup.pyinstallQuick Usage>>>fromtokenlistsimportTokenListManager>>>tlm=TokenListManager()>>>tlm.available_tokenlists()[]>>>tlm.install_tokenlist("tokens.1inch.eth")>>>tlm.available_tokenlists()['1inch']LicenseThis project is licensed under theMIT license.
Package: tokenmanager
Very simple. You have tokens for all your REST APIs. This library
manages them.What is this for?In my own work, I find myself having to code up a solution to the
problem: “I need to connect to this REST API, but I don’t want to have
the token saved in my code repo.” The solutions often involve either
sourcing bash scripts with environment variables or saving some
out-of-repo config file with the needed token.The goal of this library is to provide a dead simple solution. Create
a “dot” file (~/.tokenmanager.yml) with all of your tokens that
you can read from / write to when necessary.InstallationInstall usingpip:pip install tokenmanagerInitialize the.tokenmanager.ymlfile in your home directory:python -m tokenmanager -iUsageTo add tokens, specify them using YAML syntax:service_a: 239e896bfe6f3b7705696c8cef84832e760b689a9633c454432fc407cb7a17af
service_b: 18c553b119e52cb7d1ab93699865dab38d02d09d13b453258edf859ec075d9d9
digitalocean:
  app1: a1e1c084540b51b33af3c6b63d48ede2937c8df92f7e6e3beb1f630ac750b851
  app2: 03593464105708646cc04d847ffc81c5b7775c462f68b573f2aff5d933635e17
twilio:
  sid: WF32428983a878dv89df9xcfjn8d983298
  token: 0b51b33af3c6b63d48ede2937c8df92f
  number: '+12825552341'To access them from Python:>>> tokens = tokenmanager.get_tokens()
>>> tokens.service_a
'239e896bfe6f3b7705696c8cef84832e760b689a9633c454432fc407cb7a17af'
>>> tokens.digitalocean.app1
'a1e1c084540b51b33af3c6b63d48ede2937c8df92f7e6e3beb1f630ac750b851'To use from bash:$ python -m tokenmanager service_a
239e896bfe6f3b7705696c8cef84832e760b689a9633c454432fc407cb7a17af
$ python -m tokenmanager digitalocean.app1
a1e1c084540b51b33af3c6b63d48ede2937c8df92f7e6e3beb1f630ac750b851
Package: tokenROI
tokenROI========:Info: Console ethereum erc20 token portfolio.<br>:Author: Lukasz Czerwinski <mrowacz@gmail.com>About=====tokenROI is an tool which eaisly presents your erc20 token collectionusing data from idex stock.It converts your token balances into ethereum and also shows current roiof your investments. There is possibility to store your's configuration ongoogle drive.Installation============download the project source and do::```$ python setup.py install```Examples========Here's a basic usage examples```$ token_roi```console output for example should gives<br>![tokenroi_basic_example](https://user-images.githubusercontent.com/11421787/38460421-a2b09c78-3ab9-11e8-88f3-d132d834b48d.png)
Package: tokensjar
tokensjartokensjaris a pure Python module dedicated to generate, manipulate and interpret expressions following environment variables creation rules. This module is really useful if you need to develop a tool that can prepare the session environment variables for a specific usage.Installationtokensjaris available onPyPipython -m pip install tokensjarConceptAtokenmust be considered as a simple variable where we can attach one value (arawvalue) or multiple values (appendorprepend). The main goal is to manipulate tokens with the same strategies than environment variables.ExamplesAdd values to the tokensThere are three methods to prepare theTokensJar:add_raw_value: Define the value for the token. Only the last value added will be kept.add_prepend_value: Allow to prepend a string to the token value. Each value will be separated by the OS standard separator (':' on Unix, ';' on Windows).add_append_value: Allow to append a string to the token value. Each value will be separated by the OS standard separator (':' on Unix, ';' on Windows).importosfromtokensjarimportTokensJarjar=TokensJar(init_tokens={'PATH':os.environ['PATH'],'LD_LIBRARY_PATH':os.environ['LD_LIBRARY_PATH']})jar.add_raw_value('HELLO','Hello World!')print(jar.tokens_interpreted['HELLO'])# Output: HelloWorld!jar.add_prepend_value('PATH','MyPrependPath')print(jar.tokens_interpreted['PATH'])# Output: MyPrependPath:[...environment PATHs...]jar.add_append_value('LD_LIBRARY_PATH','MyAppendPath')print(jar.tokens_interpreted['LD_LIBRARY_PATH'])# Output: [...environment LD_LIBRARY_PATHs...]:MyAppendPathInterpret expressionsThe strength of theTokensJaris the interpretation engine behind. You can add values that refers to other tokens as much as you want. Aninterpretmethod andtokens_interpretedproperty are exposed to compute/retrieve values.fromtokensjarimportTokensJarjar=TokensJar()jar.add_raw_value('T1','Token1')jar.add_raw_value('T2','Token2')jar.add_raw_value('T12','$(T1)/$(T2)')# Note the usage of the syntax $()jar.add_raw_value('HELLO':'Hello: ')print(jar.tokens_interpreted['T12'])# Output: Token1/Token2result=jar.interpret('Joe said: $(HELLO) $(T12)')# Output: Joe said: Hello: Token1/Token2Take care to not introduce cyclic dependencies between tokens! The topological sort is handled by Eric V. SmithtoposortPython package (Link).fromtokensjarimportTokensJarjar=TokensJar()jar.add_raw_value('T1','$(T2)')jar.add_raw_value('T2','$(T1)')jar.interpret('$(T1)')# Raise toposort EXCEPTION: Cyclic dependency detected!ContributionsFeel free to open feature requests, issues or pull requests. Your help and vision are welcome!TestsTests are written followingunittestframework. Some dependencies are needed (test-requirements.txt). If you want to run tests, enter the following command at the root level of the package:python -m unittest discoverBuild the projectTo perform the build, you need to installdist-requirements.txtlist of packages, then run the following command. All the files will be located indistdirectory.python -m buildUpload the projectThe project can be uploaded usingtwine(listed indist-requirements.txt) by running the following command:python -m twine upload dist/*