Saravanan666/pypi-org huggingface dataset:
Package: token-bucket
A Token Bucket Implementation for Python Web AppsThetoken-bucketpackage provides an implementation of thetoken bucket algorithmsuitable for use in web applications for shaping or policing request
rates. This implementation does not require the use of an independent
timer thread to manage the bucket state.Compared to other rate-limiting algorithms that use a simple counter,
the token bucket algorithm provides the following advantages:The thundering herd problem is avoided since bucket capacity is
replenished gradually, rather than being immediately refilled at the
beginning of each epoch as is common with simple fixed window
counters.Burst duration can be explicitly controlledMoving window algorithms are resistant to bursting, but at the cost of
additional processing and memory overhead vs. the token bucket
algorithm which uses a simple, fast counter per key. The latter approach
does allow for bursting, but only for a controlled duration.
Package: token-count
Token CountToken Count is a command-line utility that counts the number of tokens in a text string, file, or directory, similar to the Unixwcutility. It uses the OpenAItiktokenlibrary for tokenization and is compatible with GPT-3.5-turbo or any other OpenAI model token counts.InstallationTo install Token Count, run the following command in your terminal:pipinstalltoken-countUsageToken Count has three main options:Count tokens in a text string:token-count--text"Your text here"Count tokens in a file:token-count--filepath/to/your/file.txtCount tokens in a directory (recursively):token-count--directorypath/to/your/directoryYou can provide any combination of these options. Token Count will print the token count for each input type.Additionally, you can provide any OpenAI model(gpt-4) to get token count according to the model. By default it uses "gpt-3.5-turbo".token-count--model_name"gpt-4"LicenseThis project is licensed under the MIT License.
Package: tokencrypt
This module provide cryptography compatible proxy class for using
a crypto smartcard , (such as yubi key, or PKCS#11 token) to
process the actually cryptographic workload.These are normally considered to be ‘secure’ tokens and the private
key they are normally configured so the private key is not extractable
from the token’s hardware. As a result you need to request the token
to do any cryptographic operation for which you would normally use the
private key.In this module you will find a new abstraction of cryptography’sRSAPrivateKeyclass calledRSAPrivateTokenwhich instead using
one the normal cryptography provided backend uses an hardware token
directly.Currently this means the PKCS#11 API, as provided on Linux by the
opensc-pkcs11[1]package. TokenCrypt uses the PyKCS11 library
to wrap the opensc PKCS#11 library and needs a environment variable
set so the library can be found on debian that would beexportPYKCS11LIB=/usr/lib/x86_64-linux-gnu/opensc-pkcs11.soA use case of this library would be to use a yubikey to provide Oauth
identity services to python applications which use the requests-Oauth2
library.  This is quite common if you are accessing 3rd party APIs to
cloud service systems.In this case aRSAPrivateTokencan be provided to the library in the place
of the  usualRSAPrivateKey.Many library examples show providing an PEM encoded string, but they also
work with a subclass of cryptography’sRSAPrivateKey, as shown below.ExampleimportTokenCryptfromThierAPIimportAPIConection,PrivateCredentials# Not a working example.rsa_key=TokenCrypt.RSAPrivateToken(slot=0,key=0,pin='123456')withrsa_key:credentials=PrivateCredentials(args.consumer_key,rsa_key)api=APIConection(credentials)do_something_with(api)[1]This is the package name in debian Stretch, Buster, Bullseye(so far), your
distribution may vary.Current StatusThis is an initial proof of concept release which is design to have enough
code to support working as an Oauth client. As a result the only action currently
implemented on the key is signing.But there is a lot todo, Pull requests and bug reports welcome.TODOImplementRSAPublicToken.Dynamically select the signing mechanism based to the provided
padding and hash.Implement decrypt.Implement certificate extraction.
Package: tokenease
TokenEaseTokenEase is a versatile and efficient tokenizer, designed to streamline the process of converting text into bag-of-words (BoW) vectors for natural language processing tasks. With its customizable options, TokenEase provides a smooth and seamless experience for developers and researchers alike. It is built on top of the popularspaCyand CountVectorizer fromscikit-learnlibraries, and is designed to be easy to use and integrate into your existing projects. It only supports English language at the moment. If you are interested in contributing to the project, please make a pull request or open an issue.InstallationInstallation using pip:pipinstalltokeneaseInstallation from source (easy to do if you want to contribute or modify the code!):Ideally create a virtual environment for Python 3.10+ and install poetry. Then install tokenease with poetry:poetryinstallUsageHere's a simple usage guide for the Pipe class that you can include in your README.md:ThePipeclass is used to preprocess text data for natural language processing tasks. It provides a pipeline that can perform various transformations on the text data, such as removing accents, converting to lowercase, removing stop words, and tokenizing the text.Here's a basic example of how to use thePipeclass:fromtokeneaseimportPipe# Initialize the pipeline with the desired optionspipe=Pipe(strip_accents=True,lowercase=True,remove_stop_words=True)# Fit the pipeline to your data and transform it into a bag of words representationbow,docs=pipe.fit_transform(my_data)# Transform new data using the fitted pipelinenew_bow,new_docs=pipe.transform(new_data)# bow is a numpy array of shape (n_samples, n_features).# docs is a list of strings, where each string is a document in the tokenized form with separator.Saving and Loading the PipelineYou can save the state of the pipeline to a file and load it later. This is useful if you want to reuse the same pipeline across multiple sessions or scripts.Here's how you can do it:# Save the pipelinepipe.save('my_pipeline.joblib')# Load the pipelineloaded_pipe=Pipe.from_pretrained('my_pipeline.joblib')Accessing the VocabularyAfter fitting the pipeline to your data, you can access the vocabulary (i.e., the set of unique tokens {tokens: indices}) like this:# Get the vocabularyvocab=pipe.vocabularyPlease note that this is a basic guide and you might need to adjust it based on the specific features and requirements of your project.
Package: tokenfinderbot
Token Pool BotTokenPoolBot is a simple bot for scanning new ERC-20 token pools created and filtering them based on the time created, liquidity and market cap.
On each run it shows potential good token pools based on the filters.Disclaimer: These markets are volatile and this bot just filters the new pools. If you want to take action on the output of this code, you have to check each pool by yourself and do your own researchHow to useInstall the packagepip install tokenfinderbotBasic usage(default settings)fromtokenfinderbot.tokenfinderbotimportTokenBot# bot instancebot=TokenBot()# run with default settingsbot.run()Advanced usage(edit settings): SeeSettingsSectionfromtokenfinderbot.tokenfinderbotimportTokenBot# bot instancebot=TokenBot()# get bot settingssettings=bot.get_settings()# change bot settingssettings.time_filter.hour=6# time filter hour editsettings.liq_mc_filter.min_liq=1000# liquidity/market cap filter edit for min. liquiditysettings.db_name="test"# change json db filename# set new settingsbot.set_settings(settings)# run with new settingsbot.run()Notify on telegram(You should have a telegram bot token and a chat id which the bot is a member)fromtokenfinderbot.tokenfinderbotimportTokenBot# bot instancebot=TokenBot()# get bot settingssettings=bot.get_settings()# change bot telegram settingssettings.telegram.notify=Truesettings.telegram.bot_token="YOUR TELEGRAM BOT TOKEN"# your telegram bot tokensettings.telegram.chat_id="xxxxxxxxxx"# chat id for chat, group or channel which the bot is a memeber of# set new settingsbot.set_settings(settings)# run with new settingsbot.run()SettingsBot settings:DB Name(settings.db_name): Name of the json file used as database. This project makes use oftinydbfor database.(default: "db")Time Filter(settings.time_filter): Time filter is used to accept only the pools that have been created less than x hours and y minutes ago. This filter setting includes:Hours(settings.time_filter.hours): Hours(default: 12)Minutes(settings.time_filter.minutes): Minutes(default: 0)Liquidity/Market Cap Filter(settings.liq_mc_filter): Filter based on liquidity and market cap which includes three settings:MarketCap/Liquidity Ratio(settings.liq_mc_filter.mc_liq_ratio): The minimum accepted ratio of marketcap/liquidity for the pool to be accepted. for example, 2 means the market cap must be at least 2 times the liquidity.(default: 1.5)Min. Market Cap(settings.liq_mc_filter.min_mc): The minimum accepted marketcap in USD(default: 100000)Min. Liquidity(settings.liq_mc_filter.min_liq): The minimum accepted liquidity in USD(default: 1000)Update Interval(settings.update_interval): Time interval between each update of the database and calling APIs. Its unit is minutes(default: 15)Telegram(settings.telegram): Telegram bot settings to get notifications on telegram:Notify(settings.telegram.notify): This should beTruein order to activate the telegram notifications.(default: False)Bot Token(settings.telegram.bot_token): Your telegram bot token(default: "")Chat ID(settings.telegram.chat_id): Your telegram chat id. The bot should be a member of this chat. it can be a private chat, group or channel(default: "")
Package: token-fuse
一.生成和校验通用的token安装和引入Jwt模块pip install token-fuse

from token_fuse import Jwt实例化jwt = Jwt()组织payload必选参数：
    1. exp 过期时间 ，now > exp 则失效
可选参数：
    1. uid：用户辨识用户的唯一id，建议[uid, user]任选一种作为参数
    2. iss:签发人
    3. iat：签发时间戳

payload = {'user': 'admin', 'iss': 'django', 'iat': 1642147838, 'exp': 1642152838}生成tokentoken = jwt.token(payload)校验tokenresult = jwt.auth(token)打印校验结果如校验成功返回payload，失败则返回一个布尔为False的AuthFailed对象，打印该对象可查看具体错误信息print(result)二. 生成和校验带“熔断”功能的token在有些项目中，业务要求限制允许token登录的最大数量，本模块利用redis记录token，达到限制登录数量的目的。
引入JwtFuse模块from token_fuse import JwtFuse配置Django setting.py# 如果JWT_CONF缺省，max_login=1
# redis=dict(host='127.0.0.1', port=6379, db=0)
JWT_CONF = {
    'max_login': 3,
    'redis': dict(host='127.0.0.1', port=6379, db=0)
}实例化jwt_fuse = JwtFuse()组织payload必选参数有:
    1.exp 过期时间 ，now > exp 则失效
    2.iat：签发时间戳 。
    3.uid：用户辨识用户的唯一id，建议[uid, user]任选一种作为参数名都可以
可选参数有：
    1.iss:签发人，在同一台服务器中如果存在多个项目（多项目共用了Redis数据池），
    如果这些项目不是共享token，那么在JwtRange().auth()时为了防止token串号，可以设置iss用于区分。

payload = {'user': 'admin', 'iss': 'django', 'iat': 1642147838, 'exp': 1642152838}生成token如果传入了request参数，则默认生成带有设备信息的token，
然后根据request中的User-Agent将访问的请求区分为移动和PC两种设备类型。
再将设备类型{‘ua’:’m’}或{‘ua’:’p’}更新到payload中，
最后使用带有设备类型标记的payload生成token。
不同设备类型不共享max_login,也就是说max_login=1时，ua:p和ua:m的设备能同时登录访问。token = jwt_fuse.token(payload, request=None)校验token校验通过的token会被保存在redis中，保存时间3600秒。保存格式：uid:[token1,token2…]，auth函数只会通过最新签发时间的max_len条token.result = jwt_fuse.auth(token)打印校验结果如校验成功返回payload，失败则返回一个布尔为False的AuthFailed对象，打印该对象可查看具体错误信息print(result)
Package: tokengen
Configure Azure CLI from CLI or Environment Variables# 1. Authenticate with Azure CLI Creds from Terminalazlogin# 2. Authenticate with Environmental variablesexportAZURE_CLIENT_ID="*******"exportAZURE_TENANT_ID="*******"exportAZURE_CLIENT_SECRET="*******"# Make sure Role has Azurekeyvault read/write permissionsAuthenticate with Gitlab Registry# In order not to be passing flags for the gitlab credentials, export them once on your terminal sessionexportusername="*******"exportemail="*******"exportpassword="*******"How to Setup# Create virtual environment -- Optionalpython-mvenvvenv&&sourcevenv/bin/activate# Install via pippipinstalltokengen# Check commands and subcommandstokengen-h# Available subcommandstokengengenerate# To create 4 Tokens at Oncetokengenbridgetoken# To optionally create Bridgetoken laterHow to generate tokens# Create and Store 4 Tokens in Azurekeyvault at ONCE -- Optional(bridgetoken)# 1. AIFI_API_TOKEN and PEPPER Token as retailere>-cloud-oasis-api-retailer-token# 2. KEYCLOAK credentials as <retailer>-cloud-keycloak-admin# 3. RETAILER_TOKEN and oasis-api-token secret as <retailer>-cloud-oasis-api-token# 4. RETAILER_BRIDGE_TOKEN as <store-id>-retailer-bridge-tokenGenerate 4 Tokens at ONCE# --btoken is optional if you want to generate a Bridgetoken, else just elimimate it from the --flagstokengengenerate--vault<vault_name>\--vault_url<vault_url>\--secret<secret_name>\--domain<store-domain-prefix>\--btokenronaldo-express-003# To Use Default credentials of keyvault, Runtokengengenerate--secret<retailer_name_example_zucchini>--domainholtebarGenerate Bridgetoken separately# tokengen bridgetoken --btoken <store-id>tokengenbridgetoken--btokenumari-express-0001Why Python Secrets?# From Python DocsThesecretsmoduleisusedforgeneratingcryptographicallystrongrandomnumberssuitableformanagingdatasuchaspasswords,accountauthentication,securitytokens,andrelatedsecrets.

Inparticularly,secretsshouldbeusedinpreferencetothedefaultpseudo-randomnumbergeneratorintherandommodule,whichisdesignedformodellingandsimulation,notsecurityorcryptography.To Contribute, Setup Environment# create venvpython3-mvenvvenv# Activate the venvsourcevenv/bin/activate# Install packagespipinstall-rrequirements.txt# Switch to developer modepython3setup.pydevelop# To test with your Azure keyvault, pass these flags to bypass default flags set for keyvault--vault
--vault_url
Package: token-generator
TTRPG Token GerenatorDescriçãoUm gerador simples e de facil uso de imagens para tokens de TTRPG. Utilizados em sites de mesa digital como oRoll20.Utilizaçãogenerate-token:Criar token baseado em uma imagegenerate-token<path/to/image>remove-background:Remover o fundo de uma imagemremove-background<path/to/image>Repositorio:token-generator
Package: tokenhmac
Token HMACLike JWT but more faster as it possible.About 4x times faster thanPyJWT.Installation$ pip install tokenhmacExamplesimporttokenhmacKEY="Secret Key"token=tokenhmac.TokenHMAC(KEY)value="Message"encoded=token.encode(value.encode("utf-8"))print(encoded)# b'TWVzc2FnZQ==.aKA0izCttsIOAWvblvsB5xdY17DlaOzlMVeKG1f0u0o='token.decode(encoded).decode("utf-8")# "Message"importtokenhmacKEY="Secret Key"token=tokenhmac.TokenHMAC(KEY)value=123encoded=token.encode(value.to_bytes(8,"big"))print(encoded)# b'AAAAAAAAAHs=.FAqptzNnojMDx_bfoH6PKTBWP7tFpo4CQyGseVUM9vA='int.from_bytes(token.decode(encoded),"big")# 123importtokenhmacimportjsonKEY="Secret Key"token=tokenhmac.TokenHMAC(KEY)value={"id":123,"msg":"Message"}encoded=token.encode(json.dumps(value).encode("utf-8"))print(encoded)# b'eyJpZCI6MTIzLCJtc2ciOiJNZXNzYWdlIn0=.NlnFLagfGDSn5XmSBmVnmDgqAsBApGx6GFXuvYtjPb4='json.loads(token.decode(encoded).decode("utf-8"))# {"id": 123, "msg": "Message"}Licensetokenhmacis offered under the MIT license.
Package: token-information
Discord token information librarySteps to install:pip3 install token_informationCommands:token_information.user(token)token_information.payment(token)token_information.username(token)token_information.userid(token)token_information.phonenumber(token)token_information.avatar(token)token_information.email(token)token_information.auth(token)token_information.flags(token)token_information.language(token)token_information.verified(token)token_information.login(token)token_information.creation_date(token)token_information.nitro(token)
Package: tokeniser
TokeniserTokeniseris a lightweight Python package designed for simple and efficient token counting in text. It uses regular expressions to identify tokens, providing a straightforward approach to tokenization without relying on complex NLP models.InstallationTo installTokeniser, you can use pip:pipinstalltokeniserUsageTokeniseris easy to use in your Python scripts. Here's a basic example:importtokenisertext="Hello, World!"token_count=tokeniser.estimate_tokens(text)print(f"Number of tokens:{token_count}")This package is ideal for scenarios where a simple token count is needed, without the overhead of more complex NLP tools.FeaturesSimple and efficient token counting using regular expressions.Lightweight with no dependencies on large NLP models or frameworks.Versatile for use in various text processing tasks.ContributingContributions, issues, and feature requests are welcome! Feel free to check theissues page.LicenseThis project is licensed under theMIT License.
Package: tokenizador
Table of ContentsObjetivoArgparserObjetivoO objetivo deste TPC era a criação de um package python que recebesse como input ficheiros textos e fosse capaz de os tokenizar. Passando como argumento alguns regex costumizados transformá-los em tokens que iriam aparecer no texto final.Por fim era ainda objetivo a disponibilização do módulo feito num repositório como o pypi ou o testpypiEra ainda possível adicionar tokens costumizáveis com o argumento –abbreviations, sendo estas definidas em formato jsonArgparserusage: Tokenizor [-h] [--files [FILES ...]] [--chapter CHAPTER] [--pagebreak PAGEBREAK] [--output [OUTPUT ...]]
                 [--append] [--abbreviations ABBREVIATIONS]

Tokenize a file given as input

options:
  -h, --help            show this help message and exit
  --files [FILES ...], -f [FILES ...]
                        Define the file that is supposed to be tokenized
  --chapter CHAPTER, -cap CHAPTER
                        Define the regex for a Chapter
  --pagebreak PAGEBREAK, -pb PAGEBREAK
                        Define the regex for a page break
  --output [OUTPUT ...], -o [OUTPUT ...]
                        Define the output files to be written. The output files will map to the same order of
                        input files. If number of output files is less than the number of input files then the
                        other files will be written to the stdin.
  --append, -a          Define if the output is supposed to be appended
  --abbreviations ABBREVIATIONS, -abbr ABBREVIATIONS
                        Define the path to a json with a json that maps the regex of the abbreviation with the
                        token wanted.
Package: tokenization-layer
This is a package for a concept about a tokenization algorithm that is a neural network layer, training as part of a model trying to solve some NLP task, to make tokens that are best for the task. You can find more information on theGitHub repository.This package mainly consists of theTokenizationLayer, which is atf.keraslayer doing the described above. However it also contains initializers for the layer's parameter, a function for one-hot encoding a string as letters, and an Embedding layer, that doesn't have to be the first layer in the network unlike the official keras version.Documentation for this package ishere.
Package: tokenize-all
Tokenize-AllTokenize blocks of code in Python. Used bymanim-code-blocksto syntax highlight blocks of code.Example Usagefromtokenize_allimportJavatokens=Java.tokenize("""public class Main {public static void main(String[] args) {System.out.println("Hello world!");}}""")fortokenintokens:print(token)Output:>> Token[ type = keyword, value = public, start = 4, end = 10 ]
>> Token[ type = keyword, value = class, start = 11, end = 16 ]
>> Token[ type = class name, value = Main, start = 17, end = 21 ]
>> Token[ type = left brace, value = {, start = 22, end = 23 ]
>> Token[ type = keyword, value = public, start = 31, end = 37 ]
>> Token[ type = keyword, value = static, start = 38, end = 44 ]
>> Token[ type = keyword, value = void, start = 45, end = 49 ]
>> Token[ type = function, value = main, start = 50, end = 54 ]
>> Token[ type = left parentheses, value = (, start = 54, end = >> 55 ]
>> Token[ type = class name, value = String, start = 55, end = >> 61 ]
>> Token[ type = left bracket, value = [, start = 61, end = 62 ]
>> Token[ type = right bracket, value = ], start = 62, end = 63 ]
>> Token[ type = identifier, value = args, start = 64, end = 68 ]
>> Token[ type = right parentheses, value = ), start = 68, end = >> 69 ]
>> Token[ type = left brace, value = {, start = 70, end = 71 ]
>> Token[ type = class name, value = System, start = 83, end = >> 89 ]
>> Token[ type = dot, value = ., start = 89, end = 90 ]
>> Token[ type = identifier, value = out, start = 90, end = 93 ]
>> Token[ type = dot, value = ., start = 93, end = 94 ]
>> Token[ type = function, value = println, start = 94, end = >> 101 ]
>> Token[ type = left parentheses, value = (, start = 101, end = >> 102 ]
>> Token[ type = string, value = "Hello world!", start = 102, >> end = 116 ]
>> Token[ type = right parentheses, value = ), start = 116, end >> = 117 ]
>> Token[ type = semicolon, value = ;, start = 117, end = 118 ]
>> Token[ type = right brace, value = }, start = 126, end = 127 ]
>> Token[ type = right brace, value = }, start = 131, end = 132 ]
Package: tokenize-all-code
Tokenize All CodeThis is a clone of the original code that was foundhere. That repo has since disappeared, so I have cloned it here for archival and development purposes.Tokenize blocks of code in Python. Used bymanim-code-blocksto syntax highlight blocks of code.Example Usagefromtokenize_allimportJavatokens=Java.tokenize("""public class Main {public static void main(String[] args) {System.out.println("Hello world!");}}""")fortokenintokens:print(token)Output:>> Token[ type = keyword, value = public, start = 4, end = 10 ]
>> Token[ type = keyword, value = class, start = 11, end = 16 ]
>> Token[ type = class name, value = Main, start = 17, end = 21 ]
>> Token[ type = left brace, value = {, start = 22, end = 23 ]
>> Token[ type = keyword, value = public, start = 31, end = 37 ]
>> Token[ type = keyword, value = static, start = 38, end = 44 ]
>> Token[ type = keyword, value = void, start = 45, end = 49 ]
>> Token[ type = function, value = main, start = 50, end = 54 ]
>> Token[ type = left parentheses, value = (, start = 54, end = >> 55 ]
>> Token[ type = class name, value = String, start = 55, end = >> 61 ]
>> Token[ type = left bracket, value = [, start = 61, end = 62 ]
>> Token[ type = right bracket, value = ], start = 62, end = 63 ]
>> Token[ type = identifier, value = args, start = 64, end = 68 ]
>> Token[ type = right parentheses, value = ), start = 68, end = >> 69 ]
>> Token[ type = left brace, value = {, start = 70, end = 71 ]
>> Token[ type = class name, value = System, start = 83, end = >> 89 ]
>> Token[ type = dot, value = ., start = 89, end = 90 ]
>> Token[ type = identifier, value = out, start = 90, end = 93 ]
>> Token[ type = dot, value = ., start = 93, end = 94 ]
>> Token[ type = function, value = println, start = 94, end = >> 101 ]
>> Token[ type = left parentheses, value = (, start = 101, end = >> 102 ]
>> Token[ type = string, value = "Hello world!", start = 102, >> end = 116 ]
>> Token[ type = right parentheses, value = ), start = 116, end >> = 117 ]
>> Token[ type = semicolon, value = ;, start = 117, end = 118 ]
>> Token[ type = right brace, value = }, start = 126, end = 127 ]
>> Token[ type = right brace, value = }, start = 131, end = 132 ]Building and PublishingRequires Python 3.9 or later.pipinstalltwinebuild
python-mbuild
twineuploaddist/*