Saravanan666/pypi-org huggingface dataset:
Package: deep-nccl-wrapper
Deep-NCCL-WrapperDeep-NCCL-Wrapper is a wrapper forDeepNCCLwhich Optimized primitives for inter-GPU communication on Aliyun machines.IntroductionDeep-NCCL is an AI-Accelerator communication framework for NVIDIA-NCCL.
It implements optimized all-reduce, all-gather, reduce, broadcast, reduce-scatter, all-to-all, as well as any send/receive based communication pattern.
It has been optimized to achieve high bandwidth on aliyun machines using PCIe, NVLink, NVswitch, as well as networking using InfiniBand Verbs, eRDMA or TCP/IP sockets.InstallTo install Deep NCCL on the system, create a package then install it as root as follow two methods:method1: rpm/deb (Recommended)# Centos:wgethttps://aiacc.oss-accelerate.aliyuncs.com/nccl/rpm/deep-nccl-2.0.1.rpm
rpm-ideep-nccl-2.0.1.rpm# Ubuntu:wgethttps://aiacc.oss-accelerate.aliyuncs.com/nccl/deb/deep-nccl-2.0.1.deb
dpkg-ideep-nccl-2.0.1.debmethod2: python-pypipipinstalldeep-nccl-wrapperUsageAfter install deep-nccl package, you need do nothing to change code!EnvironmentAIACC_FASTTUNING: Enable Fasttuning for LLMs, default=1 is to enable.NCCL_AIACC_ALLREDUCE_DISABLE: Disable allreduce algo, default=0 is to enable.NCCL_AIACC_ALLGATHER_DISABLE: Disable allgather algo, default=0 is to enable.NCCL_AIACC_REDUCE_SCATTER_DISABLE: Disable reduce_scatter algo, default=0 is to enable.AIACC_UPDATE_ALGO_DISABLE: Disable update aiacc nccl algo from aiacc-sql-server, default=0 is to enable.PerformanceDeep-NCCL can speedup the nccl performance on aliyun EGS(GPU machine), for example instance type 'ecs.ebmgn7ex.32xlarge' is A100 x 8 GPU and using network eRdma.GPU(EGS)CollectiveNodesNetworkSpeedup(nccl-tests)A100 x 8all_gather2-10VPC/eRdma30%+A100 x 8reduce_scatter2-10VPC/eRdma30%+A100 x 8all_reduce2-10VPC/eRdma20%V100 x 8all_reduce2-20VPC60%+A10  x 8all_reduce1-20%CopyrightAll source code and accompanying documentation is copyright (c) 2015-2020, NVIDIA CORPORATION. All rights reserved.
All modifications are copyright (c) 2020-2024, ALIYUN CORPORATION. All rights reserved.
Package: deepneighbor
DeepNeighborEmbedding-based Retrieval for ANN Search and Recommendations!View Demo·Report Bug·Request FeatureDeepNeighbor is aHigh-level,FlexibleandExtendiblepackage for embedding-based information retrieval from user-item interaction logs. Just as the name suggested,'deep'means deep learning models to get user/item embeddings, while'neighbor'means approximate nearest neighbor search in the embedding space.It mainly has two parts : Embed step and Search step by the following codes:model = Embed(data_path); model.train()，which generates embeddings for users and items (Deep),model.search(), which looks for Approximate nearest neighbor for seed user/item (Neighbor) .InstallpipinstalldeepneighborHow To UsefromdeepneighborimportEmbedmodel=Embed(data,model='gat')model.train()model.search(seed='Louis',k=10)Input formatThe input data for theEmbed()should be a (*.csv or *.txt ) file path (e.g. '\data\data.csv')with two columns in order: 'user' and 'item'. For each user, the item are recommended to be ordered by time.Models & parameters in Embed()Word2Vecw2vFactorization MachinesfmDeep Semantic Similarity ModelSiamese Network with triple lossDeepwalkGraph convolutional networkNeural Graph Collaborative Filtering algorithmngcfMatrix factorizationmfGraph attention networkgatModel Parametersdeepwalkmodel=Embed(data,model='deepwalk')model.train(window_size=5,workers=1,iter=1dimensions=128)window_sizeSkip-gram window size.workersUse these many worker threads to train the model (=faster training with multicore machines).iterNumber of iterations (epochs) over the corpus.dimensionsDimensions for the node embeddingsgraph attention networkmodel=Embed(data,model='gat')model.train(window_size=5,learning_rate=0.01,epochs=10,dimensions=128,num_of_walks=80,beta=0.5,gamma=0.5,)window_sizeSkip-gram window size.learning_ratelearning rate for optimizing graph attention networkepochsNumber of gradient descent iterations.dimensionsDimensions for the embeddings for each node (user/item)num_of_walksNumber of random walks.betaandgammaRegularization parameter.How To Searchmodel.search(seed, k)seedThe Driver for the algorithmskNumber of Nearest Neighbors.ExamplesOpenColabto run the example with facebook data.ContactPlease contactlouiswang524@gmail.comfor collaboration or providing feedbacks.LicenseThis project is under MIT License, please seeherefor details.
Package: deep-ner
Named entity recognizer based on ELMo or BERT as feature extractor and
CRF as final classifier.The goal of this project is creation of a simple Python package with
the sklearn-like interface for solution of different named entity
recognition tasks in case number of labeled texts is very small (not
greater than several thousands). Special neural network language models
named as ELMo (Embeddings from Language Models) and BERT (Bidirectional
Encoder Representations from Transformers) ensure this possibility,
because these language model were pre-trained on large text corpora and
so they can select deep semantic features from text.
Package: deep-nlp
NLP Deep Learning FrameworkThis is a deepl learning framework for classification and seq2seq tasks.Installationpip install deep-nlpExample ProjectStructure├── data              --> containing the trainings and validation data
|   ├── train.csv     --> training dataset
|   └── val.csv       --> validation dataset
├── Experiment.py     --> containing the model and training logic
└── dataset.py        --> containing the Dataset objectDataset.pyfromtorch.utils.dataimportDatasetimportpandasaspdclassExampleDataset(Dataset):def__init__(self,split:str):self.data=pd.read_csv(f'{split}.csv')def__len__(self):returnlen(self.data)def__getitem__(self,idx):returnself.data.iloc[idx]Experiment.pyfromdeep_nlpimportExperiment,unpackfromdatasetimportExampleDatasetfromtransformersimportDistilBertTokenizerFast,DistilBertForSequenceClassificationimporttorchclassClassificationExperiment(Experiment):defget_tokenizer(self):tokenizer=DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')returntokenizerdefget_model(self):model=DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased')returnmodeldefbatch_fn(self,batch):source,target=zip(*batch)source_inp=self.tokenizer(source,padding=True,return_tensors=True)target=torch.tensor(target)returnunpack(source_inp,target)defrun_experiment():experiment=ClassificationExperiment(80,# batch size20,# number of epochsExampleDataset,gpus=-1,# use all available gpuslr=2.65e-5,weight_decay=4e-3,name='example_run'# name for mlflow)experiment.run()if__name__=='__main__':run_experiment()
Package: deepnlpf
DeepNLPF: A Framework for Integrating Linguistic Analysis and Semantic Annotation of Text Documents.The DeepNLPF it contains support for running various accurate natural language processing tools. For detailed information please visit ourofficial website.DeepNLPF has been implemented and tested using theUbuntu19.04 operating system. However, it may work on other similar linux versions or Windows and MacOS if it satisfies the dependencies on external NLP tools mentioned below "at your own risk".Install and User Guide
Package: deepobs
# DeepOBS - A Deep Learning Optimizer Benchmark Suite![DeepOBS](docs/deepobs_banner.png “DeepOBS”)[![PyPI version](https://badge.fury.io/py/deepobs.svg)](https://badge.fury.io/py/deepobs)
[![Documentation Status](https://readthedocs.org/projects/deepobs/badge/?version=stable)](https://deepobs.readthedocs.io/en/latest/?badge=stable)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)DeepOBSis a benchmarking suite that drastically simplifies, automates and
improves the evaluation of deep learning optimizers.It can evaluate the performance of new optimizers on a variety ofreal-world test problemsand automatically compare them withrealistic baselines.DeepOBS automates several steps when benchmarking deep learning optimizers:Downloading and preparing data sets.Setting up test problems consisting of contemporary data sets and realistic
deep learning architectures.Running the optimizers on multiple test problems and logging relevant
metrics.Reporting and visualization the results of the optimizer benchmark.![DeepOBS Output](docs/deepobs.jpg “DeepOBS_output”)The code for the current implementation working withTensorFlowcan be found
on [Github](https://github.com/fsschneider/DeepOBS).
A PyTorch version is currently developed (see News section below).The full documentation is available on readthedocs:https://deepobs.readthedocs.io/The paper describing DeepOBS has been accepted for ICLR 2019 and can be found
here:https://openreview.net/forum?id=rJg6ssC5Y7If you find any bugs in DeepOBS, or find it hard to use, please let us know.
We are always interested in feedback and ways to improve DeepOBS.## NewsWe are currently working on a new and improved version of DeepOBS, version 1.2.0.
It will supportPyTorchin addition to TensorFlow, has an easier interface, and
many bugs ironed out. You can find the latest version of it in [this branch](https://github.com/fsschneider/DeepOBS/tree/v1.2.0-beta0).A pre-release, version 1.2.0-beta0, will be available shortly and a full release is expected in a few weeks.Many thanks to [Aaron Bahde](https://github.com/abahde) for spearheading the developement of DeepOBS 1.2.0.## Installationpip install deepobsWe tested the package with Python 3.6 and TensorFlow version 1.12. Other
versions of Python and TensorFlow (>= 1.4.0) might work, and we plan to expand
compatibility in the future.If you want to create a local and modifiable version of DeepOBS, you can do this directly from this repo viapip install -e git+https://github.com/fsschneider/DeepOBS.git#egg=DeepOBSfor the latest stable version, orpip install -e git+https://github.com/fsschneider/DeepOBS.git@v1.2.0-beta0#egg=DeepOBSto get the preview of DeepOBS 1.2.0.Further tutorials and a suggested protocol for benchmarking deep learning
optimizers can be found onhttps://deepobs.readthedocs.io/
Package: deepoc
DeepOCDeepOC is the core ofBioModel Classifier- the python application to classify biomodels automatically using Deep Neural Network. DeepOC provides some very low level functions for classifying model based on ontology, which allow us to adapt to any other projects.Installingpip install deepocUsageFirst, you need a ground truth dataset, which is a dict of model and list of it's corresponding ontologies{
    "model_1": ["GO:00001", "GO:00003", "GO:00002"],
    "model_2: ["GO:00004", "GO:00002"]
}To generate dataset and train DNN model:ground_truth = ...
train_file = "path/to/your train csv file"
test_file = "path/to/your test csv file"
val_file = "path/to/your val csv file"
features = deepoc.build_features(ground_truth)

# Picking the first 300 features  
selected_features = [feature['feature'] for idx, feature in enumerate(features) if idx < 300]

train, test, val = deepoc.generate_dataset(ground_truth, features, classes)

# Writing dataset to file  
deepoc.write_dataset_to_file(train, train_file)  
deepoc.write_dataset_to_file(test, test_file)  
deepoc.write_dataset_to_file(val, val_file)

# Configure DNN model to use Gradient Descent optimizer, 1 hidden layer with 150 nodes, learning rate of 0.001 and dropout rate of 0.5
classifier = DeepOCClassifier(workspace, 'GD', [150], 0.001, train_file, test_file, classes, 0.5)
# Train the model with 3000 epoch, validate every 10 epochs and batch size of 16
classifier.train_dll_model(3000, 10, 16)

# Validate the result:
for record in val:
    model = record['model']
    predict_result = classifier.predict(record)  
    logger.info('Model %s: %s', model, predict_result)More examples can be found intestsfolder.Classify model based on any ontology other than Gene OntologyTo make this library work with other kind of ontology, implement theOntologyServiceaccording to your ontology and instantize your object athttps://bitbucket.org/biomodels/deepoc/src/master/deepoc/ontology/init.pyDevelopersVu TuContactLicensingBiological Model Classifier source code is distributed under the GNU Affero General Public License.Please readlicense.txtfor information on the software availability and distribution.
Package: deep-oc-client
deep-oc-clientDEEP OC Command Line Interface (DEEP OC CLI).This is a command line tool (and also a library) to interact with theDEEP-Hybrid-DataCloud Marketplace,
allowing you to browse, get information, download and execute the published
modules.Free software: Apache License 2.0Source:https://github.com/alvarolopez/deep-oc-clientBugs:https://github.com/alvarolopez/deep-oc-client/issuesDocumentation: TBDInstallationYou can install it via PyPI:pip install deep-oc-clientUsageTo list the modules you can usemodule ls:$ deep-oc module list
+-------------------------------------------+-------------------------------------------------+--------------------------------------------------------------------+------------+
| Title                                     | DockerHub container                             | url                                                                | License    |
+-------------------------------------------+-------------------------------------------------+--------------------------------------------------------------------+------------+
| DEEP OC Dogs breed detection              | deephdc/deep-oc-dogs_breed_det                  | https://github.com/deephdc/DEEP-OC-dogs_breed_det                  | MIT        |
+-------------------------------------------+-------------------------------------------------+--------------------------------------------------------------------+------------+To fetch information about a module you can usemodule show:$ deep-oc module show https://github.com/deephdc/deep-oc-dogs_breed_det
+----------------------+---------------------------------------------------------------------------+
| Field                | Value                                                                     |
+----------------------+---------------------------------------------------------------------------+
| TOSCA_template       | Yes                                                                       |
| build_status         | SUCCESS                                                                   |
| date_creation        | 2018-11-18                                                                |
| docker_registry_repo | deephdc/deep-oc-dogs_breed_det                                            |
| dockerfile_repo      | https://github.com/deephdc/DEEP-OC-dogs_breed_det                         |
| keywords             | ['docker', 'tensorflow', 'cnn']                                           |
| license              | MIT                                                                       |
| model_source_code    | https://github.com/deephdc/dogs_breed_det                                 |
| summary              | A test application to identify Dog's breed as an example for DEEPaaS API. |
| title                | DEEP OC Dogs breed detection                                              |
+----------------------+---------------------------------------------------------------------------+In order to get the complete list of commands, as well as usage details please
check the output of:deep-oc help
Package: deepoffense
DeepOffense : Multilingual Offensive Language Identification with TransformersDeepOffense provides state-of-the-art transformer models for multilingual offensive language identification.InstallationYou first need to install PyTorch. THe recommended PyTorch version is 1.5.
Please refer toPyTorch installation pageregarding the specific install command for your platform.When PyTorch has been installed, you can install from source by cloning the repository and running:gitclonehttps://github.com/TharinduDR/DeepOffense.gitcdDeepOffense
pipinstall-rrequirements.txtRun the examplesExamples are included in the repository but are not shipped with the library. Please refer theexamplesdirectory for the examples. Each directory in theexamplesfolder contains different languages.Pretrained ModelsEnglish offensive language detection pre-trained model trained with XLM-R  large model on OffensEval data can be downloaded using thislink.Once downloading it and unzipping it, they can be loaded easily. To see how to begin the training process please refer theexamplesdirectorymodel=ClassificationModel("xlmroberta","path",use_cuda=torch.cuda.is_available())CitationPlease consider citing us if you use the library.@inproceedings{ranasinghe-etal-2020-multilingual,title="Multilingual Offensive Language Identification with Cross-lingual Embeddings",author="Ranasinghe, Tharindu  andZampieri, Marcos,booktitle = "Proceedingsofthe2020ConferenceonEmpiricalMethodsinNaturalLanguageProcessing",month = novyear = "2020",}Citation for the Malayalam specific paper,@inproceedings{ranasinghe-etal-2020-wlv,title={WLV-RITatHASOC2020:OffensiveLanguageIdentificationinCode-switchedTexts},author={Ranasinghe,TharinduandZampieri,Marcos},year={2020},booktitle={ProceedingsofFIRE}}
Package: deepomatic-api
deepomatic-client-pythonDeepomaticAPI Client for Python.This client have been made in order to help you integrating our services within your apps in python.Tested on python 2.7, 3.4, 3.5, 3.6.API Documentationhttps://docs.deepomatic.com/deepomatic-api-v0-7Installationpipinstalldeepomatic-apiClientInitialize a client.
Does not make any call to the server.fromdeepomatic.api.clientimportClient# If you don't have your credentials, contact us at support@deepomatic.comclient=Client(api_key=api_key,user_agent_prefix='my-app/1.0.0')app_id is now optional but if you don't put it, you have to ensure that the api_key is a named parameter to the functionClient methodsAll client methods can be found indeepomatic/api/client.pyand detail for each type of resource is located indeepomatic/api/resources.ExamplesYou will find examples of usage indemo.py.BugsPlease send bug reports tosupport@deepomatic.comor open an issue here.
Package: deeponnxcustom
deeponnxcustom: custom ONNX and deep learningdocumentationOnnx, onnxruntime, deep learning, pytorch…python setup.py build_ext --inplaceGenerate the setup in subfolderdist:python setup.py sdistGenerate the documentation in folderdist/html:python setup.py build_sphinxRun the unit tests:python setup.py unittestsTo check style:python -m flake8 deeponnxcustom tests examplesThe functioncheckor the command linepython-mdeeponnxcustom checkchecks the module is properly installed and returns processing
time for a couple of functions or simply:import deeponnxcustom
deeponnxcustom.check()
Package: deepop
DeepOpsA Mini Deep learning library, accelerated on GPUs with PyCuda.No no no.. I havent wrote this library on tensorflow or torch, it is a standalone machine. :PImplemented backpropogations using reverse traversal, Support gradients and GPU operations, You can make a mini neural network (FNN) and use the in-house optimizer to train it on a dataset (e.g MNIST).Tip: always give your tensor a funny name! :)Note: Only for Educational Usage.Installation.pip install deepopsTensor.a = Tensor([1,2,3,4,5])
# deepop tensorAttach to a cuda device.a = Tensor([1,2,3,4,5])
a.device("gpu:0") # attach to gpu device.Check the Device.a.where
# 'cpu'Addition.a = Tensor([1.0,2.0])
print(a + a)
# GPU OperationMultiplication.a = Tensor([1.0, 2.0])
print(a.mul(a))
print(a * a)Calculate Gradients.Tensor = dp.Tensor
a1 = Tensor([1.0, 3.0, 1.0])
b1 = Tensor([7.0, 3.0, 5.0])

a2 = Tensor([4.0, 3.0, 1.0])
a3 = Tensor([3.0, 3.0, 1.0])
a4 = Tensor([7.0, 1.0, 6.0])
b2 = Tensor([1.0, 21.0, 12.0])

c = a1 * b1 + a3
d = a2 * b2 + a4
out = c * d

# backward
out.backward()

print(out.grad)
print(a1.grad)Run Tests.python -m pytest -sContribution is highly appreciated.Please contribute to my work.TODOswrite more tests...need a optimizer.support more operations.LicenseMIT
Package: deepops
PYTHON DEEPOPS MODULEThis module contains some functions for performing "deep" operations on
standard Python "compound" data structures - lists/sets/dictionaries:deepmerge()- merges two compound structures, including all sub-items,
e.g. items to lists and sets, missing keys/items to dictionaries, optionally
replacing clashing simple types.deepremoveitems()- removes items (simple types, or whole compound
structures) from within another compound data structure.deepfilter()- returns the part of one compound structure that is matched
by the other: essentially the opposite of deepremoveitems().deepdiff()- compares two compound structures and returns a tuple of items
to be removed and items to be updated: these could be passed todeepremoveitems()anddeepmerge(), respectively, to transform one into
the other (although note that they would need to bedeepcopy()ed first).deepsetdefault()- similar to dict.setdefault() except that it can set the
default at arbitrarily deep paths into a dictionary.deepget()- similar to dict.get() but can copy with arbitrarily deep paths
into an indexable structure.The module was developed and used under Python 3.4-3.7 but seems to
work OK in basic testing under 2.7.AuthorRobert Franklinrcf@mince.net, UK
Package: deepparcellation
DeepParcellation PackageDeepParcellation: fast and accurate brain MRI parcellation by deep learningContributionsThe project was initiated by Dr. Lim (abysslover) and Dr. Choi (yooha1003).The code is written by Dr. Lim at Gwangju Alzheimer's & Related Dementias (GARD) Cohort Research Center (GARD CRC), Chosun University.This research was conducted in collaborations with the following people:
Eun-Cheon Lim1, Uk-Su Choi1, Yul-Wan Sung2, Kun-Ho Lee1and Jungsoo Gim1.Gwangju Alzheimer's & Related Dementias (GARD) Cohort Research Center, Chosun University, Gwangju, Republic of KoreaKansei Fukushi Research Institute, Tohoku Fukushi University, Sendai, Miyagi, JapanThe manuscript will be available in the future.Getting StartedA step-by-step tutorial is provided in the following sections.PrerequisitesYou should install CUDA-enabled GPU cards with at least 8GB GPU memory manufactured by nVidia, e.g., Titan XP.Prepare T1-weighted MR imagesConvert MR images to Neuroimaging Informatics Technology Initiative (NIfTI) format.The parent directory name of a NIfTI file path will be used asSubject Idduring prediction.You can specify eitheran input pathof the NIfTI file orinput direcotryof many NIfTI files.Install DeepParcellationInstall AnacondaDownload an Anaconda distribution:LinkCreate a Conda environmentconda create -n deepparc python=3.8 -yInstall DeepParcellation (CPU mode)conda activate deepparc
	pip install deepparcellationInstall DeepParcellation (GPU mode)conda activate deepparc
	pip install deepparcellation
	conda install cudnn=7.6.5 -c anaconda -y
	conda install cudatoolkit=10.1.243 -c conda-forge -y
	pip uninstall tensorflow -y
	pip install tensorflow-gpu==2.2.0
	pip uninstall keras -y
	conda install keras-gpu=2.4.3 -c anaconda -yInstall DeepParcellation (Force)conda activate deepparc
	pip install --force --no-dependencies deepparcellationRun DeepParcellationconda activate deepparc
	deepparcellation -o=/tmp/test --i=./subject-0-0000/test.nii.gz
	or
	deepparcellation -o=/tmp/test --i=./NOTE:You must alwaysactivate the conda enviromentbefore running DeepParcellation if you opened anew console.You should install DeepParcellation with force parameter when all the other dependencies were manually met, i.e., you have installed tensorflow-macos, tensorflow-metal dependencies for Apple M1 chips.ContactPlease contactabysslover@gmail.comif you have any questions about DeepParcellation.
Package: deeppath
Python module to easily manipulate complex nested structuresFree software: MIT licenseDocumentation:https://deeppath.readthedocs.io.FeaturesWith~deeppath.dget, you can access data in a complex nested
structure. The nested structure should be a json-like structure, so
essentially consisting of dictionary- and list-like structures:nested_data = {
    "users": [
        {"surname": "Doe", "name": "John"},
        {"name": "Jane", "surname": "Doe"},
    ],
    "location": {
        "city": "London",
        "country": "United Kingdom",
    },
}CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.