Saravanan666/pypi-org huggingface dataset:
Package: torch-shapeguard
ShapeGuardShapeGuard allows you to very succinctly assert the expected shapes of tensors in a dynamic, einsum inspired wayTurn this:defbatch_outer_product(x,y):# x has shape (batch, x_channels)# y has shape (batch, y_channels)# return has shape (batch, x_channels, y_channels)returnx.unsqueeze(-1)*y.unsqueeze(-2)Into this:defbatch_outer_product(x,y):x.sg(("batch","x_channels"))y.sg(("batch","y_channels"))return(x.unsqueeze(-1)*y.unsqueeze(-2)).sg(("batch","x_channels","y_channels"))Installationpip install torch-shapeguardMotivationIt’s easy to make bugs in ml.
One particular rich source of bugs is due to the flexibility of the operators:a*bworks whether a and b are vectors, scalar vector, vector vector, etc.
Similarly.sum()will work regardless of the shape of your tensor.
Since we're doing optimization whatever computation we end up performing, we can probably optimize it to work reasonably, even if it's not doing what we intended.
So our algorithm might "work" even if we have bugs (just less well).
This makes bugs super hard to discover.The best way I’ve found to avoid bugs is to religiously check the shapes of all my tensors, all the time, so I end up spending a lot of time debugging and writing comments like#(bs, n_samples, z_size)all over the place.So why not algorithmically check the shapes then? Well it gets ugly fast.You have to add assertfoo.shape == (bs, n_samples, x_size)everywhere, which essentially doubles your linecount and
you have to define all your dimensional sizes (bs, etc.), which might vary across train/test, batches, etc.
So I made a small helper that makes it much nicer. I call it ShapeGuard.UsageWhen youimport shapeguard, It adds thesgmethod totorch.Tensorandtorch.distributions.Distribution.You can use thesgmethod like an assert:defforward(self,x,y):x.sg("bchw")y.sg("by")This will verify that x has 4 dimensions, y has 2 dimensions and that x and y have the same size in the first dimension 'b'.If the assert passes, the tensor is returned.
This means you can also chain it inline on results of operations:z=f(x).sg("bnz").mean(axis=1).sg("bz")If the assert fails it produces a nice error message:AssertionError: expected 'b' to be 2 but was 4If you want to verify an exact dimension you can pass an int as the shape e.g.defforward(self,x,y):x.sg(("b",1,"h","w"))y.sg("by")The special shape '*' is reserved for shapes that should not be asserted, e.g.x.sg("*chw")will assert all shapes except the first.How it worksThe first timesgis called for an unseen shape, the size of the tensor for that shape is saved in theShapeGuard.shapesglobal dict.
Subsequent calls are checked against this stored shape.You can callShapeGuard.reset(shape)to reset a specific shape.
This can be useful if e.g. your batch size varies between runs.ShapeGuard.reset()resets all shapes.
Package: torch-simivec
torch-simivec : Multi-label Embedding Training as Similarity Learning ProblemTrain an input multi-label embedding as a similarity learning problem.UsageModellingfromtorch_simivecimportSimiLossimporttorchimportnumpyasnp# init the modelmodel=SimiLoss(tokenlist_size=10,embedding_size=256,context_size=4)# create a positive & negative exampleX_pos=torch.tensor([[[0,2,9],[6,0,8],[1,3,4],[7,8,9]]])y_pos=torch.tensor([[1,4,2]])np.random.seed(42)X_neg=torch.tensor(np.random.permutation(X_pos))y_neg=torch.tensor(np.random.permutation(y_pos))# compute lossloss=model(y_pos,X_pos,y_neg,X_neg)print(loss)Trainingoptimizer=torch.optim.Adam(model.parameters(),lr=3e-4)avg_loss=.0forepochinrange(50):optimizer.zero_grad()loss=model(y_pos,X_pos,y_neg,X_neg)loss.backward()optimizer.step()avg_loss+=loss.item()if(epoch%10)==9:print(f"epoch{epoch+1}| loss:{avg_loss/10.}")avg_loss=.0AppendixInstallationThetorch-simivecgit
repois available asPyPi
packagepipinstalltorch-simivecpipinstallgit+ssh://git@github.com/ulf1/torch-simivec.gitInstall a virtual environmentpython3-mvenv.venvsource.venv/bin/activatepipinstall--upgradepippipinstall-rrequirements.txt--no-cache-dirpipinstall-rrequirements-dev.txt--no-cache-dirpipinstall-rrequirements-demo.txt--no-cache-dir(If your git repo is stored in a folder with whitespaces, then don’t use
the subfolder.venv. Use an absolute path without whitespaces.)Python commandsJupyter for the examples:jupyter labCheck syntax:flake8--ignore=F401--exclude=$(grep-v'^#'.gitignore | xargs | sed-e's//,/g')Run Unit Tests:PYTHONPATH=. pytestPublishpandocREADME.md--frommarkdown--torst-s-oREADME.rstpythonsetup.pysdisttwineupload-rpypidist/*Clean upfind.-typef-name"*.pyc"|xargsrmfind.-typed-name"__pycache__"|xargsrm-rrm-r.pytest_cacherm-r.venvSupportPleaseopen an
issuefor support.ContributingPlease contribute usingGithub
Flow. Create a branch,
add commits, andopen a pull
request.
Package: torch-simple-timing
💻  Code•Docs  📑Torch Simple TimingA simple yet versatile package to time CPU/GPU/Multi-GPU ops."I want to time operations once"That's what aClockis for"I want to time the same operations multiple times"That's what aTimeris forIn simple terms:AClockis an object (and context-manager) that will compute the ellapsed time between itsstart()(or__enter__) andstop()(or__exit__)ATimerwill internally manage clocks so that you can focus on readability and not data structuresInstallationpip install torch_simple_timingHow to useAClockfromtorch_simple_parsingimportClockimporttorcht=torch.rand(2000,2000)gpu=torch.cuda.is_available()withClock(gpu=gpu)ascontext_clock:torch.inverse(t@t.T)clock=Clock(gpu=gpu).start()torch.inverse(t@t.T)clock.stop()print(context_clock.duration)# 0.29688501358032227print(clock.duration)# 0.292896032333374More examples, including bout how to easily share data structures using astorecan be found in thedocumentation.ATimerfromtorch_simple_timingimportTimerimporttorchdevice=torch.device("cuda"iftorch.cuda.is_available()else"cpu")X=torch.rand(5000,5000,device=device)Y=torch.rand(5000,100,device=device)model=torch.nn.Linear(5000,100).to(device)optimizer=torch.optim.Adam(model.parameters())gpu=device.type=="cuda"timer=Timer(gpu=gpu)forepochinrange(10):timer.clock("epoch").start()forbinrange(50):x=X[b*100:(b+1)*100]y=Y[b*100:(b+1)*100]optimizer.zero_grad()withtimer.clock("forward",ignore=epoch>0):p=model(x)loss=torch.nn.functional.cross_entropy(p,y)withtimer.clock("backward",ignore=epoch>0):loss.backward()optimizer.step()timer.clock("epoch").stop()stats=timer.stats()# use stats for display and/or logging# wandb.summary.update(stats)print(timer.display(stats=stats,precision=5))epoch    : 0.25064 ± 0.02728 (n=10)
forward  : 0.00226 ± 0.00526 (n=50)
backward : 0.00209 ± 0.00387 (n=50)A decoratorYou can also use a decorator to time functions without much overhead in your code:fromtorch_simple_timingimporttimeit,get_global_timer,reset_global_timerimporttorch# Use the function name as the timer name@timeit(gpu=True)deftrain():x=torch.rand(1000,1000,device="cuda"iftorch.cuda.is_available()else"cpu")returntorch.inverse(x@x)# Use a custom name@timeit("test")deftest_cpu():returntorch.inverse(torch.rand(1000,1000)@torch.rand(1000,1000))if__name__=="__main__":for_inrange((epochs:=10)):train()test_cpu()timer=get_global_timer()print(timer.display())reset_global_timer()Prints:train : 0.045 ± 0.007 (n=10)
test  : 0.046         (n= 1)By default the@timeitdecodrator takes at least aname, will usegpu=Falseand use the global timer (torch_simple_timing.TIMER). You can pass your own timer with@timeit(name, timer=timer).Seein the docs.
Package: torch-simplify
SimplifySimplification of pruned models for accelerated inference.Fully supports torchvision classification models up
toversion 0.11.InstallationSimplify can be installed using pip:pip3installtorch-simplifyor if you want to run the latest version of the code, you can install from git:gitclonehttps://github.com/EIDOSlab/simplifycdsimplify
pip3install-rrequirements.txtUsageMain functionFor most scenarios the mainsimplifyfunction will suffice. This function returns the simplified model.ArgumentsThe expected arguments are:model (torch.nn.Module): Module to be simplified i.e. the PyTorch's model.x (torch.Tensor): zero-tensor of shape[1, C, N, M], same as the model usual input.bn_folding (List): List of tuple (nn.Conv2d,nn.BatchNorm2d) to be fused. If None it tries to evaluate them
given the model. DefaultNone.fuse_bn (bool): If True, fuse the conv-bn tuple.pinned_out (List): List ofnn.Moduleswhich output needs to remain of the original shape (e.g. layers related to a
residual connection with a sum operation).Minimal working exampleimporttorchfromtorchvisionimportmodelsfromsimplifyimportsimplifymodel=models.resnet18()# Apply some pruning strategy or load a pruned checkpointdummy_input=torch.zeros(1,3,224,224)# Tensor shape is that of a standard input for the given modelsimplified_model=simplify(model,dummy_input)SubmodulesThesimplifyfunction is composed of three different submodules:fuse,propagateandremove. Each module can be
used independently as needed.fuseFuses adjacent Conv (or Linear) and BatchNorm layers.propagatePropagates non-zero bias of pruned neurons to remaining neurons of the next layers.removeRemoves zeroed neurons from the architecture.CitingIf you use this software for research or application purposes, please use the following citation:@article{bragagnolo2021simplify,title={Simplify: A Python library for optimizing pruned neural networks},journal={SoftwareX},volume={17},pages={100907},year={2022},issn={2352-7110},doi={https://doi.org/10.1016/j.softx.2021.100907},url={https://www.sciencedirect.com/science/article/pii/S2352711021001576},author={Andrea Bragagnolo and Carlo Alberto Barbano},}
Package: torch-skeleton
torch_skeletonEfficient datasets and transforms for skeleton dataInstallation$pipinstalltorch_skeletonDocumentationhttps://torch-skeleton.readthedocs.io/en/latest/index.html#DatasetsDownload and load raw dataset with preprocessfromtorch_skeleton.datasetsimportNTUimporttorch_skeleton.transformsasT# dwonload ntu skeleton datasetntu=NTU(root="data",num_classes=60,eval_type="subject",split="train",transform=T.Compose([T.Denoise(),T.CenterJoint(),T.SplitFrames(),]),)x,y=ntu[0]Cache preprocessed samples to diskfromtorch_skeleton.datasetsimportDiskCache# cache preprocessing transforms to diskcache=DiskCache(root="data/NTU",dataset=dataset)x,y=cache[0]Apply augmentations to a datasetfromtorch_skeleton.datasetsimportApply# cache preprocessing transforms to diskcache=Apply(dataset=dataset,transform=T.Compose([T.SampleFrames(num_frames=20),T.RandomRotate(degrees=17),T.PadFrames(max_frames=20),]),)x,y=cache[0]Training Models usingtorch_skeletonExample Training code usingtorch_skeletonis available underexamplesSupported models:SGNLicensetorch_skeletonwas created by Chanhyuk Jung. It is licensed under the terms
of the MIT license.
Package: torchslide
TorchSlideWorks only on Python-3.6Bundles all binaries fromASAPto single packageProvides array-like interface forASAP::MultiResolutionImageUsage:importtorchslideastsslide=ts.Image('test.svs')shape:'Tuple[int]'=slide.shapescales:'Tuple[int]'=slide.scalesimage:np.ndarray=slide[:2048,:2048]# get numpy.ndarray
Package: torchslide-any
TorchSlideWorks on Python-3.6+Compiles from sourcesProvides array-like interface for reading BigTIFF/SVS filesUsage:importtorchslideastsslide=ts.Image('test.svs')shape:'Tuple[int]'=slide.shapescales:'Tuple[int]'=slide.scalesimage:np.ndarray=slide[:2048,:2048]# get numpy.ndarrayInstallationCurrentlytorchslideis only supported under 64-bit Windows and Linux machines.
Compilation on other architectures should be relatively straightforward as no OS-specific libraries or headers are used.
The easiest way to install the software is to download package fromPyPI.CompilationTo compile the code yourself, some prerequesites are required.
First, we usesetuptools >= 40as our build system and Microsoft Build Tools or GCC as the compiler.
The software depends on numerous third-party libraries:libtiff (http://www.libtiff.org/)libjpeg (http://libjpeg.sourceforge.net/)DCMTK (http://dicom.offis.de/dcmtk.php.en)OpenSlide (http://openslide.org/)zlib (http://www.zlib.net/)To help developers compile this software themselves we provide the necesarry binaries (Visual Studio 2017, 64-bit) for all third party libraries on Windows.
If you want to provide the packages yourself, there are no are no strict version requirements, except for libtiff (4.0.1 and higher).
On Linux all packages can be installed through the package manager on Ubuntu-derived systems (tested on Ubuntu and Kubuntu 16.04 LTS).To compile the source code yourself, first make sure all third-party libraries are installed.
Package: torchsm
torchsmpytorch implementation of theStigmergic Memoryas presented in the paperUsing stigmergy as a computational memory in the design of recurrent neural networks.You can use this package toeaslyintegrate our model into existing onesYou can safelymixnative pytorch Modules with ours.Butdo not forgettoreset()them before starting every new time sequenceImplementing ourproposed architecture to solve MNISTbecomes as easy as:importtorchimporttorchsmnet=torchsm.Sequential(torchsm.RecurrentStigmergicMemoryLayer(28,15,hidden_layers=1,hidden_dim=20),torch.nn.Linear(15,10),torch.nn.PReLU(),torch.nn.Linear(10,10),torch.nn.PReLU())You can train the time-unfolded model by computing the loss function on the desired temporal outputoptimizer=torch.optim.Adam(net.parameters(),lr=0.001)loss_fn=torch.nn.MSELoss()foriinrange(0,N):forX,Yinzip(dataset_X,dataset_Y):net.reset()out=Noneforiinrange(0,X.shape[1]):out=net(torch.tensor(X[:,i],dtype=torch.float32))loss=loss_fn(out,Y)optimizer.zero_grad()loss.backward()optimizer.step()Does it support batch inputs?Yes! The inputs have to be batchedfortinrange(0,num_ticks):batch_out[0],batch_out[1],...=net(torch.tensor([batch_in[0][t],batch_in[1][t],...]))Can it run on CUDA?Yes and as you will expect from a pytorch Module!You just need to call theto(device)method on a model to move it in the GPU memorydevice=torch.device("cuda")net=net.to(device)net(torch.tensor(...,device=device))Documentationtorchsm.SequentialWrapper oftorch.nn.Sequentialthat adds thereset()method and forward the call to eachtorchsm.BaseLayerchild.If you want to use aSequentialContaierto build your models with one or more torchsm's layers you have to usetorchsm.Sequentialinstead oftorch.nn.Sequentialin order to be able toreset()them.torchsm.StigmergicMemoryLayerThis layer has two hidden ANNs with the layer's inputs as inputs and which outputs respectively determine the marks and ticks of a multi-monodimensional stigmergic space.torchsm.RecurrentStigmergicMemoryLayerThis layer is a StigmergicMemoryLayer which output is normalized by a linear layer and recurrently forwarded as input to the two hidden ANNsCitingWe can't wait to see what you will build with torchsm!When you will publish your work you can use this BibTex to cite us :)@article{galatolo_snn
,	author	= {Galatolo, Federico A and Cimino, Mario GCA and Vaglini, Gigliola}
,	title	= {Using stigmergy as a computational memory in the design of recurrent neural networks}
,	journal	= {ICPRAM 2019}
,	year	= {2019}
,	pages	= {}
}ContributingThis code is released under GNU/GPLv3 so feel free to fork it and submit your changes, every PR helps.If you need help using it or for any question please reach me atfederico.galatolo@ing.unipi.itor on Telegram@galatolo
Package: torch-snake
SnakeBased on"Neural Networks Fail to Learn Periodic Functions and How to Fix It"by Liu Ziyin, Tilman Hartwig, Masahito UedaThis is a PyTorch implementation of thesnakeactivation function from the paper - or at least Ithinkit is, no affiliation with the authors, use at your own risk, etc., etc.A few variations of the function are discussed in the paper, this package implements:Snake isperiodic, butalsomonotonic. To see howsnakebehaves for a range of x given various choices ofa, watch this video:InstallationTwo methods:Usingpip,pip install torch-snakeTo install from source, first clone this repository.  Then, from the main repo folder, runpython setup.py installUsageFairly easy reallyfrom snake.activations import Snake.  TheSnakeconstructor(code here)has an optionallearnableparameter alpha which defaults to 1.  The authors of the paper find values between 5 and 50 work quite well for "known-periodic" data, while for better results with non-periodic data, you should choose a small value like 0.2.  The constructor also takes analpha_learnableparameter which defaults toTrue, so that you can disable "learnability" if your experiments so require.Sample codeThere's a notebook, still quite rough -example.ipynb.  Early indications are that good choices of hyperparameters are quite important for best results (though snake's own parameter trains quite readily).AcknowledgementsThis code probably wouldn't have gotten written if it hadn't been forAlexandra Deisandher excellent article.  It has also benefitted hugely from generous contributions byFederico Berto.
Package: torchsnapshot
TorchSnapshot (Beta Release)A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind.InstallRequires Python >= 3.7 and PyTorch >= 1.12From pip:# Stablepipinstalltorchsnapshot# Nightlypipinstall--pretorchsnapshot-nightlyFrom source:gitclonehttps://github.com/pytorch/torchsnapshotcdtorchsnapshot
pipinstall-rrequirements.txt
pythonsetup.pyinstallWhy TorchSnapshotPerformanceTorchSnapshot provides a fast checkpointing implementation employing various optimizations, including zero-copy serialization for most tensor types, overlapped device-to-host copy and storage I/O, parallelized storage I/O.TorchSnapshot greatly speeds up checkpointing for DistributedDataParallel workloads by distributing the write load across all ranks (benchmark).When host memory is abundant, TorchSnapshot allows training to resume before all storage I/O completes, reducing the time blocked by checkpoint saving.Memory UsageTorchSnapshot's memory usage adapts to the host's available resources, greatly reducing the chance of out-of-memory issues when saving and loading checkpoints.TorchSnapshot supports efficient random access to individual objects within a snapshot, even when the snapshot is stored in a cloud object storage.UsabilitySimple APIs that are consistent between distributed and non-distributed workloads.Out of the box integration with commonly used cloud object storage systems.Automatic resharding (elasticity) on world size change for supported workloads (more details).SecuritySecure tensor serialization without pickle dependency [WIP].Getting StartedfromtorchsnapshotimportSnapshot# Taking a snapshotapp_state={"model":model,"optimizer":optimizer}snapshot=Snapshot.take(path="/path/to/snapshot",app_state=app_state)# Restoring from a snapshotsnapshot.restore(app_state=app_state)See thedocumentationfor more details.Licensetorchsnapshot is BSD licensed, as found in theLICENSEfile.
Package: torchsnapshot-nightly
TorchSnapshot (Beta Release)A performant, memory-efficient checkpointing library for PyTorch applications, designed with large, complex distributed workloads in mind.InstallRequires Python >= 3.8 and PyTorch >= 2.0.0From pip:# Stablepipinstalltorchsnapshot# Or, using condacondainstall-cconda-forgetorchsnapshot# Nightlypipinstall--pretorchsnapshot-nightlyFrom source:gitclonehttps://github.com/pytorch/torchsnapshotcdtorchsnapshot
pipinstall-rrequirements.txt
pythonsetup.pyinstallWhy TorchSnapshotPerformanceTorchSnapshot provides a fast checkpointing implementation employing various optimizations, including zero-copy serialization for most tensor types, overlapped device-to-host copy and storage I/O, parallelized storage I/O.TorchSnapshot greatly speeds up checkpointing for DistributedDataParallel workloads by distributing the write load across all ranks (benchmark).When host memory is abundant, TorchSnapshot allows training to resume before all storage I/O completes, reducing the time blocked by checkpoint saving.Memory UsageTorchSnapshot's memory usage adapts to the host's available resources, greatly reducing the chance of out-of-memory issues when saving and loading checkpoints.TorchSnapshot supports efficient random access to individual objects within a snapshot, even when the snapshot is stored in a cloud object storage.UsabilitySimple APIs that are consistent between distributed and non-distributed workloads.Out of the box integration with commonly used cloud object storage systems.Automatic resharding (elasticity) on world size change for supported workloads (more details).SecuritySecure tensor serialization without pickle dependency [WIP].Getting StartedfromtorchsnapshotimportSnapshot# Taking a snapshotapp_state={"model":model,"optimizer":optimizer}snapshot=Snapshot.take(path="/path/to/snapshot",app_state=app_state)# Restoring from a snapshotsnapshot.restore(app_state=app_state)See thedocumentationfor more details.Licensetorchsnapshot is BSD licensed, as found in theLICENSEfile.
Package: torch-soft
torch_soft is a high level implementatio of pytorch for some of claasification implementation with ease...featuresTensorNet currently supports the following featuresModel architecturesResNet18A custom model called naiveresnetModel utilitiesLoss functionsCross entropy lossnll_lossOptimizersStochastic Gradient DescentRegularizersL1 regularizationL2 regularizationLR SchedulersStep LRReduce LR on PlateauOne Cycle PolicyLR Range TestModel training and validationDatasets (data is is returned via data loaders)MNISTCIFAR10TinyImageNetData AugmentationResizePaddingRandom CropHorizontal FlipVertical FlipGaussian BlurRandom RotationCutOutGradCAM and GradCAM++ (Gradient-weighted Class Activation Map)Result Analysis ToolsPlotting changes in validation accuracy and loss during model trainingDisplaying correct and incorrect predictions of a trained modelPlotting images in a batch for visualizationPlotting gradcam outputsHow to UseFor examples on how to use torch_soft, refer to theexamplesdirectory.Dependenciestorch_soft has the following third-party dependenciestorchtorchvisiontorchsummarytqdmmatplotlibalbumentationsopencv-python
Package: torchsolver
torchsolverA pytorch based deep learning solver framework.installpipinstalltorchsolverexampleimporttorchfromtorchimportnn,optimfromtorchvision.datasetsimportMNISTfromtorchvision.transformsimport*fromtorchsolver.moduleimportModulefromtorchsolver.metricsimportaccuracyclassLeNet(nn.Module):def__init__(self,classes_num):super(LeNet,self).__init__()self.conv1=nn.Conv2d(1,32,5)self.pool1=nn.MaxPool2d(2,stride=2)self.conv2=nn.Conv2d(32,64,5)self.pool2=nn.MaxPool2d(2,stride=2)self.act=nn.ReLU()self.fc1=nn.Linear(1024,512)self.dropout=nn.Dropout(0.5)self.out=nn.Linear(512,classes_num)defforward(self,x):x=self.pool1(self.act(self.conv1(x)))x=self.pool2(self.act(self.conv2(x)))x=torch.flatten(x,start_dim=1)x=self.fc1(x)x=self.dropout(x)x=self.out(x)x=torch.softmax(x,dim=-1)returnxclassMnistSolver(Module):def__init__(self,**kwargs):super(MnistSolver,self).__init__(**kwargs)self.model=LeNet(10)self.loss=nn.CrossEntropyLoss()self.optimizer=optim.Adam(self.model.parameters())ifself.num_device>1:self.model=torch.nn.DataParallel(self.model)defforward(self,img,label):pred=self.model(img)acc=accuracy(pred,label)ifself.training:loss=self.loss(pred,label)returnloss,{"loss":loss,"acc":acc}else:returnacc,{}if__name__=='__main__':train_data=MNIST("data",train=True,transform=ToTensor())val_data=MNIST("data",train=False,transform=ToTensor())MnistSolver(batch_size=128).fit(train_data=train_data,val_data=val_data)
Package: torch-sparse-adamw
# Pytorch Sparse AdamW
This repository contains the sparse version of AdamW optimizer.The SparseAdamW optimizer behaves like AdamW optimizer, but updates only the statistics for gradients which are computed, in the same way as SparseAdam optimizer. The optimizer can only be used on modules, which produce sparse gradients, e.g., nn.Embedding.## Install
Install by running:`bash pip installtorch-sparse-adamw`