Saravanan666/pypi-org huggingface dataset:
Package: transformer-builder
transformer-builderMake your own transformers with ease.Transformers have become a popular choice for a wide range of Natural Language Processing (NLP) and deep learning tasks.
The transformer-builder package allows you to create custom transformer models with ease, providing flexibility and
modularity for your deep learning projects.FeaturesBuild custom transformer models with a user-friendly and flexible interface.Configurable encoder and decoder blocks with support for custom self-attention mechanisms.Encapsulated self-attention blocks that adapt to your specific use case.Open-source and customizable to fit your project's requirements.InstallationYou can install transformer-builder using pip:pipinstalltransformer-builderUsageHere's an example of how to use Transformer Builder to create a custom model:importtorchfromtorchimportnnfromtransformer_builder.attentionimportSelfAttention,MultiHeadAttentionfromtransformer_builder.layersimportResidualConnectionvocab_size=16_000embedding_dim=512num_heads=8d_head=embedding_dim//num_headsvocab_size=16_000embedding_dim=512num_heads=4num_blocks=3d_head=embedding_dim//num_headsblocks=[MultiHeadAttention(layer_before=nn.Linear(embedding_dim,embedding_dim),self_attention_heads=[SelfAttention(q_architecture=nn.Linear(embedding_dim,d_head),# Default: nn.Identityk_architecture=nn.Linear(embedding_dim,d_head),v_architecture=nn.Linear(embedding_dim,d_head),),SelfAttention(# This will calculate scaled dot product attention of original inputs# And pass the result to the linear layerlayer_after=nn.Linear(embedding_dim,d_head),),SelfAttention(layer_after=nn.Linear(embedding_dim,d_head),),SelfAttention(# Now some exotic attention architecturelayer_before=SelfAttention(),# The default value for self_attention_heads is single default headlayer_after=MultiHeadAttention(layer_after=nn.Linear(embedding_dim,d_head),))])for_inrange(num_blocks)]gpt=nn.Sequential(# nn.Embedding(vocab_size, embedding_dim), for simplicity, we will use random embeddings# ResidualConnection will add original input to the output of the module and apply normalization*[ResidualConnection(module=multi_head_attention,normalization=nn.LayerNorm(embedding_dim))formulti_head_attentioninblocks],)gpt(torch.randn(8,embedding_dim))CustomizationWith transformer-builder, you can customize each aspect of your blocks individually,
allowing for fine-grained control over your model's architecture.
The example above demonstrates how to configure the self-attention layer,
layer normalization, and linear layers.
You can go crazy and create encoder inside decoder inside self-attention!ContributingIf you would like to contribute to this project, please follow ourcontribution guidelines.Support and FeedbackIf you have questions, encounter issues, or have feedback, please open an issue on ourGitHub repository.AcknowledgmentsThis project was inspired by the need for a flexible and customizable API for creating
decoder blocks in deep learning models.AuthorMrKekovichLicenseThis project is licensed under theBSD-3-ClauseLicense.
See theLICENSEfile for details.
Package: transformer-discord-notifier
A Discord Notifier to send progress updates, params and results to a Discord channel.Free software: MIT licenseInstallationpip install transformer-discord-notifierYou can also install the in-development version with:pip install https://github.com/Querela/python-transformer-discord-notifier/archive/master.zipDocumentationhttps://python-transformer-discord-notifier.readthedocs.io/git clone https://github.com/Querela/python-transformer-discord-notifier.git
cd python-transformer-discord-notifier
sphinx-build -b html docs dist/docsDevelopmentTo run all the tests run:toxNote, to combine the coverage data from all the tox environments run:Windowsset PYTEST_ADDOPTS=--cov-append
toxOtherPYTEST_ADDOPTS=--cov-append toxChangelog0.x.0 (WIP)ignore linkcheck with version tag (if tags have not been pushed it will fail)Blocking message deletion?0.5.0 (2021-02-04)Add dynamic experiment channel creation.TODO: update docs and tests with better examples for experiment channels.0.4.5 (2021-02-04)Wrap common errors, like 5xx Discord Gateway errors, to allow uninterrupted training.Add python3.10 to tests / github workflows.0.4.4 (2020-12-22)Github Actions - tox tests0.4.3 (2020-12-18)Github Actions - pypi publishing0.4.2 (2020-12-18)Add travis build jobs.Add coveralls coverage statistics.0.4.1 (2020-12-17)Reintroduce tests withpytestandtox.Add simple tests fortransformer_discord_notifier.discord.DiscordClient.Add tests fortransformer_discord_notifier.transformers.DiscordProgressCallback.0.3.1 (2020-12-17)Let Discord bot gracefully handle initialization failures.Let transformer callback handler handle invalid configs gracefully, to simply exit.Better handling of edge cases of Discord client login.0.3.0 (2020-12-16)Add (private) scripts (make venv, run checks).Update usage docs.Extend / rewrite discord client methods.Reuse existingtqdmtransformers.trainer_callback.ProgressCallbackfor progress tracking.Fancy aggregation of prediction runs, split train progress into epochs.0.2.1 (2020-12-15)Correctsetup.pyvalidation.Add (private) distribution/docs build scripts.0.2.0 (2020-12-15)Refactor blocking discord code intotransformer_discord_notifier.discordsubmodule.Fix behaviour for__del__with refactoring, so it work as intended.Improve documentation fortransformer_discord_notifier.discordmodule.0.1.0 (2020-12-11)First release on PyPI.First working version, tested manually.Cleaned up skeleton files.Updated docs.0.0.0 (2020-12-10)Initial code skeleton.
Package: transformer-embeddings
Transformer EmbeddingsThis library simplifies and streamlines the usage of encoder transformer models supported byHuggingFace'stransformerslibrary(model hubor local) to generate embeddings for string inputs, similar to the waysentence-transformersdoes.Please note that starting with v4, we have dropped support for Python 3.7. If you need to use this library with Python 3.7, the latest compatible release isversion 3.1.0.Why use this over HuggingFace'stransformersorsentence-transformers?Under the hood, we take care of:Can be used with any model on the HF model hub, with sensible defaults for inference.Setting the PyTorch model toevalmode.Usingno_grad()when doing the forward pass.Batching, and returning back output in the format produced by HF transformers.Padding / truncating to model defaults.Moving to and from GPUs if available.InstallationYou can installTransformer EmbeddingsviapipfromPyPI:$pipinstalltransformer-embeddingsUsagefromtransformer_embeddingsimportTransformerEmbeddingstransformer=TransformerEmbeddings("model_name")If you have a previously instantiatedmodeland / ortokenizer, you can pass that in.transformer=TransformerEmbeddings(model=model,tokenizer=tokenizer)transformer=TransformerEmbeddings(model_name="model_name",model=model)ortransformer=TransformerEmbeddings(model_name="model_name",tokenizer=tokenizer)Note:Themodel_nameshould be included if only 1 of model or tokenizer are passed in.EmbeddingsTo get output embeddings:embeddings=transformer.encode(["Lorem ipsum dolor sit amet","consectetur adipiscing elit","sed do eiusmod tempor incididunt","ut labore et dolore magna aliqua."])embeddings.outputPooled OutputTo get pooled outputs:fromtransformer_embeddingsimportTransformerEmbeddings,mean_poolingtransformer=TransformerEmbeddings("model_name",return_output=False,pooling_fn=mean_pooling)embeddings=transformer.encode(["Lorem ipsum dolor sit amet","consectetur adipiscing elit","sed do eiusmod tempor incididunt","ut labore et dolore magna aliqua."])embeddings.pooledExporting the ModelOnce you are done testing and training the model, it can be exported into a single tarball:fromtransformer_embeddingsimportTransformerEmbeddingstransformer=TransformerEmbeddings("model_name")transformer.export(additional_files=["/path/to/other/files/to/include/in/tarball.pickle"])This tarball can also be uploaded to S3, but requires installing the S3 extras (pip install transformer-embeddings[s3]). And then using:fromtransformer_embeddingsimportTransformerEmbeddingstransformer=TransformerEmbeddings("model_name")transformer.export(additional_files=["/path/to/other/files/to/include/in/tarball.pickle"],s3_path="s3://bucket/models/model-name/date-version/",)ContributingContributions are very welcome. To learn more, see theContributor Guide.LicenseDistributed under the terms of theApache 2.0 license,Transformer Embeddingsis free and open source software.IssuesIf you encounter any problems, pleasefile an issuealong with a detailed description.CreditsThis project was partly generated from@cjolowicz'sHypermodern Python Cookiecuttertemplate.
Package: transformer-encoder
Transformer EncoderThis repo provides an easy-to-use interface of transformer encoder. You can use it as a general sequence feature extractor and incorporate it in 
your model.ExamplesQuickstartimporttorchfromtransformer_encoderimportTransformerEncoderfromtransformer_encoder.utilsimportPositionalEncoding# Modelencoder=TransformerEncoder(d_model=512,d_ff=2048,n_heads=8,n_layers=6,dropout=0.1)# Input embedsinput_embeds=torch.nn.Embedding(num_embeddings=6,embedding_dim=512)pe_embeds=PositionalEncoding(d_model=512,dropout=0.1,max_len=5)encoder_input=torch.nn.Sequential(input_embeds,pe_embeds)# Input data (zero-padding)batch_seqs=torch.tensor([[1,2,3,4,5],[1,2,3,0,0]],dtype=torch.long)mask=batch_seqs.ne(0)# Run modelout=encoder(encoder_input(batch_seqs),mask)Using the built-in warming up optimizerimporttorch.optimasoptimfromtransformer_encoder.utilsimportWarmupOptimizermodel=...base_optimizer=optim.Adam(model.parameters(),lr=1e-3)optimizer=WarmupOptimizer(base_optimizer,d_model=512,scale_factor=1,warmup_steps=100)optimizer.zero_grad()loss=...loss.backward()optimizer.step()Install from PyPIRequirespython 3.5+,pytorch 1.0.0+pip install transformer_encoderAPItransformer_encoder.TransformerEncoder(d_model, d_ff, n_heads=1, n_layers=1, dropout=0.1)d_model: dimension of each word vectord_ff: hidden dimension of feed forward layern_heads: number of heads in self-attention (defaults to 1)n_layers: number of stacked layers of encoder (defaults to 1)dropout: dropout rate (defaults to 0.1)transformer_encoder.TransformerEncoder.forward(x, mask)x (~torch.FloatTensor): shape(batch_size, max_seq_len, d_model)mask (~torch.ByteTensor): shape(batch_size, max_seq_len)transformer_encoder.utils.PositionalEncoding(d_model, dropout=0.1, max_len=5000)d_model: same as TransformerEncoderdropout: dropout rate (defaults to 0.1)max_len: max sequence length (defaults to 5000)transformer_encoder.utils.PositionalEncoding.forward(x)x (~torch.FloatTensor): shape(batch_size, max_seq_len, d_model)transformer_encoder.utils.WarmupOptimizer(base_optimizer, d_model, scale_factor, warmup_steps)base_optimizer (~torch.optim.Optimzier): e.g. adam optimzierd_model: equals d_model in TransformerEncoderscale_factor: scale factor of learning ratewarmup_steps: warming up stepsContributionAny contributions are welcome!
Package: transformer-implementations
Transformer ImplementationsTransformer Implementations and some examples with themImplemented:Vanilla TransformerViT - Vision TransformersDeiT - Data efficient image TransformersBERT - Bidirectional Encoder Representations from TransformersGPT - Generative Pre-trained TransformerInstallationPyPi$pipinstalltransformer-implementationsorpythonsetup.pybuild
pythonsetup.pyinstallExampleInnotebooksdirectory there is a notebook on how to use each of these models for their intented use; such as image classification for Vision Transformer (ViT) and others.
Check them out!fromtransformer_package.modelsimportViTimage_size=28# Model Parameterschannel_size=1patch_size=7embed_size=512num_heads=8classes=10num_layers=3hidden_size=256dropout=0.2model=ViT(image_size,channel_size,patch_size,embed_size,num_heads,classes,num_layers,hidden_size,dropout=dropout).to(DEVICE)prediction=model(image_tensor)Language Translationfrom "Attention is All You Need":https://arxiv.org/pdf/1706.03762.pdfModels trained with Implementation:Multi30k - German to EnglishMulti-class Image Classification with Vision Transformers (ViT)from "An Image is Worth 16x16 words: Transformers for image recognition at scale":https://arxiv.org/pdf/2010.11929v1.pdfModels trained with Implementation:MNIST - Grayscale ImagesCIFAR10 - MultiChannel ImagesNote: ViT will not perform great on small datasetsMulti-class Image Classification with Data-efficient image Transformers (DeiT)from "Training data-efficient image transformers & distillation through attention":https://arxiv.org/pdf/2012.12877v1.pdfModels trained with Implementation:Pretraining Teacher model for DistillationCIFAR10 - Low Res Images
Package: transformers-collection
transformers-collectionA collection of transformer models built  using huggingface for various tasks. Training done using pytorch lightning.Datasets, models and tokenizers from hugging face.Goal: Get familiar with huggingface and pytorch lightning ecosystems.Get startedTrain Models using the libraryTo train models, install using pip:pip install transformers_collectioncheck installation:transformers-collection versionClone project and modify codeTo play around with the code clone the repo:git clone git@github.com:aadhithya/transformers-collection.gitInstall poetry:pip install poetryIntsall dependencies:poetry installNote:poetry installwill create a new venv.Note:poetry/pip installinstalls CPU version of pytorch if not available, please make sure to install CUDA version if needed.Train a modelCreate the yaml config file for the model (see configs/sentiment-clf.yml for example).train model using:transformers-collection train /path/to/config.ymlFor a list of supported models, see section Supported Models.Supported Models / TaskThe following models are planned:ModelDatasetStatusCheckpointSentiment/Emotion Classificationemotion✅TBDText Summarization🗓️ PlannedTBD
Package: transformers-crf
Tranformers-CRFFor training BERT-CRF models with the huggingface Transformers libarynstalationgitclonehttps://bitbucket.org/avisourgente/transformers_crf.gitcdtransformers_crfpipinstall-e.Train exampleTrain script is in examples/run_ner.py
It follows the api ofhttps://github.com/huggingface/transformers/blob/main/examples/pytorch/token-classification/run_ner.pyNew args:--learning_rate_ner LEARNING_RATE_NER
                Custom initial learning rate for the CRF and Linear layers on AdamW. (default: None)
    --weight_decay_ner WEIGHT_DECAY_NER
                Custom weight decay for the CRF and Linear layers on AdamW. (default: None)
    --use_crf  
                Will enable to use CRF layer. (default: False)
    --no_constrain_crf
                Set to not to constrain crf outputs to labeling scheme (default: False)
    --break_docs_to_max_length [BREAK_DOCS_TO_MAX_LENGTH]
                Whether to chunck docs into sentences with the max seq length of tokenizer. (default: False)
    --convert_to_iobes [CONVERT_TO_IOBES]
                Convert a IOB2 input to IOBES. (default: False)Example:pythonrun_ner.py\--model_name_or_pathneuralmind/bert-base-portuguese-cased\--dataset_nameeduagarcia/portuguese_benchmark\--dataset_config_nameharem-default\--per_device_train_batch_size16\--per_device_eval_batch_size16\--num_train_epochs15\--learning_rate5e-5\--do_train\--do_eval\--do_predict\--evaluation_strategysteps\--eval_steps500\--output_dir/workspace/models/test-transformers-crf\--max_seq_length128\--break_docs_to_max_length\--overwrite_output_dir\--learning_rate_ner5e-3\--convert_to_iobes\--use_crfUsage exampleimporttorchfromtransformers_crfimportCRFTokenizer,AutoModelForEmbedderCRFTokenClassificationmodel_path="./model"device=torch.device('cpu')tokenizer=CRFTokenizer.from_pretrained(model_path)model=AutoModelForEmbedderCRFTokenClassification.from_pretrained(model_path).to(device)tokens=[["Esse","é","um","exemplo"],["Esse","é","um","segundo","exemplo"]]batch=tokenizer(tokens,max_length=512).to(device)output=model(**batch,reorder=True)predicts_id=output.predicts.detach().cpu().numpy()preds=[[model.config.id2label[p]forpinpred_seq][:len(token_seq)]for(pred_seq,token_seq)inzip(predicts_id,tokens)]
Package: transformers-domain-adaptation
Transformers Domain AdaptationDocumentation•Colab GuideThis toolkit improves the performance of HuggingFace transformer models on downstream NLP tasks,
by domain-adapting models to the target domain of said NLP tasks (e.g. BERT -> LawBERT).The overall Domain Adaptation framework can be broken down into three phases:Data SelectionSelect a relevant subset of documents from the in-domain corpus that is likely to be beneficial for domain pre-training (see below)Vocabulary AugmentationExtending the vocabulary of the transformer model with domain specific-terminologyDomain Pre-TrainingContinued pre-training of transformer model on the in-domain corpus to learn linguistic nuances of the target domainAfter a model is domain-adapted, it can be fine-tuned on the downstream NLP task of choice, like any pre-trained transformer model.ComponentsThis toolkit provides two classes,DataSelectorandVocabAugmentor, to simplify the Data Selection and Vocabulary Augmentation steps respectively.InstallationThis package was developed on Python 3.6+ and can be downloaded usingpip:pip install transformers-domain-adaptationFeaturesCompatible with the HuggingFace ecosystem:transformers 4.xtokenizersdatasetsUsagePlease refer to our Colab guide!ResultsTODO
Package: transformers-keras
transformers-keras基于tf.keras的Transformers系列模型实现。所有的Model都是keras模型，可以直接用于训练模型、评估模型或者导出模型用于部署。在线文档：transformers-keras文档本库功能预览：加载各种预训练模型的权重掩码语言模型(Masked Language Model)解决方案句向量(Sentence Embedding)的解决方案抽取式问答任务(Question Answering)的解决方案序列分类任务(Sequence Classification)的解决方案序列标注任务(Token Classification)的解决方案知识蒸馏(Knowledge Distillation)的框架以上任务的数据处理、模型构建都完成了，可以一键开始训练。这里有两个例子：使用BERT微调文本分类任务使用BERT微调问答任务更多使用方法和介绍，请查看在线文档：transformers-keras文档联系方式由于二维码过期限制，加微信讨论群请先加我微信邀请加入。微信: luozhouyang0528邮箱:zhouyang.luo@gmail.com
Package: transformer-srl
Semantic Role Lableing with BERTSemantic Role Labeling based onAllenNLP implementationofShi et al, 2019. Can be trained using both PropBank andVerbAatlasinventories and implements also the predicate disambiguation task, in addition to arguments identification and disambiguation.How to useInstall the librarypipinstalltransformer-srlDownload the pretrained modelsrl_bert_base_conll2012.tar.gzfromhere.FileModelVersionF1 ArgumentF1 Predicatesrl_bert_base_conll2012.tar.gzbert-base-cased2.4.686.095.5CLIecho'{"sentence": "Did Uriah honestly think he could beat the game in under three hours?"}'|\allennlppredictpath/to/srl_bert_base_conll2012.tar.gz---include-packagetransformer_srlInside Python Codefromtransformer_srlimportdataset_readers,models,predictorspredictor=predictors.SrlTransformersPredictor.from_path("path/to/srl_bert_base_conll2012.tar.gz, "transformer_srl")predictor.predict(sentence="Did Uriah honestly think he could beat the game in under three hours?")InfosLanguage Model: BERTDataset: CoNLL 2012Results with VerbAtlasWithbert-base-cased:# Dev set
- F1 arguments 87.6
- F1 predicates 95.5
# Test set
- F1 arguments x
- F1 predicates xWithbert-base-multilingual-cased:# Dev set
- F1 arguments 86.2
- F1 predicates 94.2
# Test set
- F1 arguments 86.1
- F1 predicates 94.9To-DosWorks with both PropBank and VerbAtlas (infer inventory from dataset reader)Compatibility with all models from Huggingface's Transformers.
- Now works only with models that accept 1 as token type idPredicate identification (without using spacy)
Package: transformers-stream-generator
transformers-stream-generatorDescriptionThis is a text generation method which returns a generator, streaming out each token in real-time during inference, based on Huggingface/Transformers.Web DemooriginalstreamInstallationpipinstalltransformers-stream-generatorUsagejust add two lines of code before your original codefromtransformers_stream_generatorimportinit_stream_supportinit_stream_support()adddo_stream=Trueinmodel.generatefunction and keepdo_sample=True, then you can get a generatorgenerator=model.generate(input_ids,do_stream=True,do_sample=True)fortokeningenerator:word=tokenizer.decode(token)print(word)Examplerun python scriptexampleby gpt2run webexampleby gpt2 and test in clientexample
Package: transformers-visualizer
Transformers visualizerExplain your 🤗 transformers without effort!Transformers visualizer is a python package designed to work with the🤗 transformerspackage. Given amodeland atokenizer, this package supports multiple ways to explain your model by plotting its internal behavior.This package is mostly based on theCaptumtutorials[1][2].Installationpipinstalltransformers-visualizerQuickstartLet's define a model, a tokenizer and a text input for the following examples.fromtransformersimportAutoModel,AutoTokenizermodel_name="bert-base-uncased"model=AutoModel.from_pretrained(model_name)tokenizer=AutoTokenizer.from_pretrained(model_name)text="The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder."VisualizersAttention matrices of a specific layerfromtransformers_visualizerimportTokenToTokenAttentionsvisualizer=TokenToTokenAttentions(model,tokenizer)visualizer(text)Instead of using__call__function, you can use thecomputemethod. Both work in place,computemethod allows chaining method.plotmethod accept a layer index as parameter to specify which part of your model you want to plot. By default, the last layer is plotted.importmatplotlib.pyplotaspltvisualizer.plot(layer_index=6)plt.savefig("token_to_token.jpg")Attention matrices normalized across head axisYou can specify theorderused intorch.linalg.normin__call__andcomputemethods. By default, an L2 norm is applied.fromtransformers_visualizerimportTokenToTokenNormalizedAttentionsvisualizer=TokenToTokenNormalizedAttentions(model,tokenizer)visualizer.compute(text).plot()Plottingplotmethod accept to skip special tokens with the parameterskip_special_tokens, by default it's set toFalse.You can use the following imports to use plotting functions directly.fromtransformers_visualizer.plottingimportplot_token_to_token,plot_token_to_token_specific_dimensionThese functions or theplotmethod of a visualizer can use the following parameters.figsize (Tuple[int, int]): Figsize of the plot. Defaults to (20, 20).ticks_fontsize (int): Ticks fontsize. Defaults to 7.title_fontsize (int): Title fontsize. Defaults to 9.cmap (str): Colormap. Defaults to "viridis".colorbar (bool): Display colorbars. Defaults to True.Upcoming featuresAdd an option to mask special tokens.Add an option to specify head/layer indices to plot.Add other plotting backends such as Plotly, Bokeh, Altair.Implement other visualizers such asvector norm.References[1]Captum's BERT tutorial (part 1)[2]Captum's BERT tutorial (part 2)AcknowledgementsTransformers Interpretfor the idea of this project.
Package: transformer-tools
Transformer-ToolsThis project is maintained by theOne Conversationteam ofDeutsche Telekom AG.LicensingCopyright (c) 2021 Philip May, Deutsche Telekom AGCopyright (c) 2021 Philip MayLicensed under theMIT License(the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License by reviewing the fileLICENSEin the repository.