Saravanan666/pypi-org huggingface dataset:
Package: pytorch-partial-tagger
pytorch-partial-taggerpytorch-partial-taggeris a Python library for building a sequence tagger, specifically for the common NLP task Named Entity Recognition, with a partially annotated dataset in PyTorch.
You can build your own tagger using a distantly-supervised dataset obtained from unlabled text data and a dictionary that maps surface names to their entity type.
The algorithm of this library is based on Effland and Collins. (2021).UsageImport all dependencies first:importtorchfromsequence_labelimportSequenceLabelfrompartial_tagger.metricimportMetricfrompartial_tagger.utilsimportcreate_trainerPrepare your own datasets. Each item of dataset must have a pair of a string and a sequence label.
A string represents text that you want to assign a label, which is defined astextbelow.
A sequence label represent a set of a character-based tag, which has a start, a length, and a label, which are defined aslabelbelow.
A start represents a position in the text where a tag starts.
A length represents a distance in the text between the beginning of a tag and the end of a tag.
A label represents what you want to assign to a span of the text defined by a start and a length.text="Tokyo is the capital of Japan."label=SequenceLabel.from_dict(tags=[{"start":0,"end":5,"label":"LOC"},# Tag for Tokyo{"start":24,"end":29,"label":"LOC"},# Tag for Japan],size=len(text),)train_dataset=[(text,label),...]validation_dataset=[...]test_dataset=[...]Here, you will train your tagger and evaluate its performance.
You will train it through an instance ofTrainer, which you get by callingcreate_trainer.
After a training, you will get an instance ofRecognizerwhich predicts character-based tags from given texts.
You will evaluate the performance of your tagger using an instance ofMetricas follows.device=torch.device("cuda")trainer=create_trainer()recognizer=trainer(train_dataset,validation_dataset,device)texts,ground_truths=zip(*test_dataset)batch_size=15predictions=recognizer(texts,batch_size,device)metric=Metric()metric(predictions,ground_truths)print(metric.get_scores())# Display F1-score, Precision, RecallInstallationpipinstallpytorch-partial-taggerDocumentationFor details about thepytorch-partial-taggerAPI,  see thedocumentation.ReferencesThomas Effland and Michael Collins. 2021.Partially Supervised Named Entity Recognition via the Expected Entity Ratio Loss.Transactions of the Association for Computational Linguistics, 9:1320–1335.
Package: pytorch-perf
PyTorch PerfInstallpip install -U pytorch-perfTo verify a successful installation:python -c "from torchperf import info; info.show()"Usageimporttorchfromtorchperfimportperf,infoinfo.show()N=100x=torch.rand(N,N,device="cuda")results=[]repeats=100@perf(o=results,n=repeats)defmul(x,y):returnx*yz=mul(x,x)print(results[:5])print("avg:",sum(results)/len(results))@perf(n=repeats)defmul(x,y):returnx*yz=mul(x,x)
Package: pytorch-pipeline
PyTorch Pipeline: Simple ETL Pipeline for PyTorchPyTorch Pipeline is a simple ETL framework for PyTorch.
It is an alternative totf.datain TensorFlowRequirementsPython 3.6+PyTorch 1.2+InstallationTo install PyTorch Pipeline:pipinstallpytorch_pipeilneBasic Usageimportpytorch_pipeilneasppd=pp.TextDataset('/path/to/your/text')d.shuffle(buffer_size=100).batch(batch_size=10).first()Usage with PyTorchfromtorch.utils.dataimportDataLoaderimportpytorch_pipeilneasppd=pp.Dataset(range(1_000)).parallel().shuffle(100).batch(10)loader=DataLoader(d,num_workers=4,collate_fn=lambdax:x)forxinloader:...Usage with LineFlowYou can use PyTorch Pipeline with pre-defined datasets inLineFlow:fromtorch.utils.dataimportDataLoaderfromlineflow.datasets.wikitextimportcached_get_wikitextimportpytorch_pipeilneasppdataset=cached_get_wikitext('wikitext-2')# Preprocessing datasettrain_data=pp.Dataset(dataset['train'])\.flat_map(lambdax:x.split()+['<eos>'])\.window(35)\.parallel()\.shuffle(64*100)\.batch(64)# Iterating datasetloader=DataLoader(train_data,num_workers=4,collate_fn=lambdax:x)forxinloader:...
Package: pytorch-polkadots
OverviewThis project provides the ability to construct a simulated image dataset
on the fly with different characteristics. While it is possible to simulate
datasets that models can be trained against successfully, it is also useful to
adjust dataset parameters to yields datasets that much more challenging to train
against.InstallationWith poetrypoetryinstallWith pippipinstallpytorch-polkadotsFrom cloned repopipinstall-rrequirements.txtTestsbash
export PYTHONTEST=.
pytestExample imageYou can see an image generated by the code with:pythonmain.pyThis will display the following image:PyTorch datasetThe code provides a pytorch dataset that you can use in benchmarks.classPolkaDotDataset(Dataset)
Package: pytorch-pqrnn
EnvironmentNote: Because of recent pytorch change (>=1.7), it is not possible to run a QRNN layer without messing up the environment. Seehttps://github.com/salesforce/pytorch-qrnn/issues/29for details.pipinstall-rrequirements.txtIf you want to use a QRNN layer, please follow the instructionshereto installpython-qrnnfirst with  downgradedtorch <= 1.4.UsageUsage:run.py[OPTIONS]Options:--task[yelp2|yelp5|toxic][default:yelp5]--bINTEGER[default:128]--dINTEGER[default:96]--num_layersINTEGER[default:2]--batch_sizeINTEGER[default:512]--dropoutFLOAT[default:0.5]--lrFLOAT[default:0.001]--nheadINTEGER[default:4]--rnn_type[LSTM|GRU|QRNN|Transformer][default:GRU]--data_pathTEXT--helpShowthismessageandexit.Datasetsyelp2(polarity): it will be downloaded w/ datasets(huggingface)yelp5:json fileshould be downloaded to intodata/toxic:datasetshould be downloaded and unzipped to intodata/Example: Yelp Polaritypython -W ignore run.py --task yelp2 --b 128 --d 64 --num_layers 4Benchmarks(not optimized)ModelModel SizeYelp Polarity (error rate)Yelp-5 (accuracy)Civil Comments (mean auroc)CommandPQRNN (this repo)78K6.370.4TODO--b 128 --d 64 --num_layers 4 --rnn_type QRNNPRNN (this repo)90K5.570.795.57--b 128 --d 64 --num_layers 1 --rnn_type GRUPTransformer (this repo)617K10.86886.5--b 128 --d 64 --num_layers 1 --rnn_type Transformer --nhead 2PRADO1175K65.9BERT335M1.8170.5898.8562PaperBest Kaggle SubmissionCreditstensorflowPowered bypytorch-lightningandgrid.ai