Saravanan666/pypi-org huggingface dataset:
Package: nchash
A fast checksum function for netCDF filesA fast hashing function for netCDF files.True hashing or checksum functions take longer the larger the file size, and hence don’t scale
well for very large files.The intention of this code was to create something that would be able to detect changes
in files as a “first pass” in a hierarchy of checksums. Full md5 or sha style checksums would
still be required to robustly identify files.The specific use case is verifying input files in ocean simulation codes, which are run, checkpointed
and then re-run in a continuous cycle. This allows for a quick integrity check on resubmission, which
if failed can trigger a slower but more rigorous full checksum comparison.The code hashes a dump of the netCDF header, the size and the modification time of the file
(internal hdf5 modification time in the case of netCDF4 formatted files).This hash is unlikely to be robust over long time periods, as there may well be changes to the
way ncdump outputs the header. But in the use-case envisaged this would trigger a more exhaustive
check, which if successful, could then regenerate a new nchash for fast checking.Over short to medium timescales, and certainly within the time taken to re-run the simulation this
would be robust to changes. Particularly so for netCDF4 files which have an internal hdf5 modfication
time stamp. In addition, netCDF4 files that are compressed will likely change size with even small
changes to variables in the file.Testing on an HPC system with fast disk access hashing is independent of size and takes 0.5-0.8s. Testing
has included file sizes up to 100GB.InstallConda install:conda install -c coecms nchashPip install (into a virtual environment):pip install nchashUseDevelopDevelopment install:git checkout https://github.com/aidanheerdegen/nchash
cd nchash
conda env create -f conda/dev-environment.yml
source activate nchash-dev
pip install -e '.[dev]'Thedev-environment.ymlfile is for speeding up installs and installing
packages unavailable on pypi,requirements.txtis the source of truth for
dependencies.Run tests:py.test
Package: nchr
nchrPackages that use nchrbytetypefuzstrinhumnbkitnistnymstrDeveloper GuideSetup# create conda environment$mambaenvcreate-fenv.yml# update conda environment$mambaenvupdate-nnchr--fileenv.ymlInstallpipinstall-e.# install from pypipipinstallnchrnbdev# activate conda environment$condaactivatenchr# make sure the nchr package is installed in development mode$pipinstall-e.# make changes under nbs/ directory# ...# compile to have changes apply to the nchr package$nbdev_preparePublishing# publish to pypi$nbdev_pypi# publish to conda$nbdev_conda--build_args'-c conda-forge'$nbdev_conda--mambabuild--build_args'-c conda-forge -c dsm-72'UsageInstallationInstall latest from the GitHubrepository:$pipinstallgit+https://github.com/dsm-72/nchr.gitor fromconda$condainstall-cdsm-72nchror frompypi$pipinstallnchrDocumentationDocumentation can be found hosted on GitHubrepositorypages. Additionally you can find
package manager specific guidelines oncondaandpypirespectively.
Package: nci
Copyright (c) 2019 Justus SchockPermission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.Description: # Python Template RepositoryPlatform: UNKNOWN
Requires-Python: >=3.7
Description-Content-Type: text/markdown
Package: nci-cidc-ngs-pipeline-api
CIDC NGS Pipeline APIOverviewThis repository serves as an interface between the CIDC and Bioinformatics teams to determine specifications and documentation related to NGS pipelines.Repository structure:.
├── README.md
├── cidc_ngs_pipeline_api
│   ├── output_API.schema.json
│   ├── rna
│   │   ├── rna.md => ../../docs/rna.md
│   │   ├── rna_config.schema.json
│   │   ├── rna_output_API.json
│   │   └── imgs
│   |       └── RIMA.png
│   ├── atacseq
│   │   ├── atacseq.md => ../../docs/atacseq.md
│   │   ├── atacseq_output_API.json
│   │   └── imgs
│   |       └── atacseq.png
│   ├── tcr
│   │   ├── tcr.md => ../../docs/tcr.md
│   │   └── imgs
│   |       └── TCRseq.png
│   └── wes
│       ├── wes.md => ../../docs/wes.md
│       ├── wes_config.schema.json
│       ├── wes_output_API.json
│       ├── wes_tumor_only_output_API.json
│       ├── wes_output_API.py
│       └── imgs
│          └── wes.png
├── tests
│   └── test_apis.py
├── requirements.dev.txt
├── requirements.txt
├── MANIFEST.in
├── CHANGELOG.md
├── setup.py
└── .github
    └── workflows
       └── ci.ymlcidc_ngs_pipeline_api moduleTheoutput_API.schema.jsonfile defines the schema structure:filter_group: Filter under which the file would appear during faceted search. It is the GCS-URI top-level hierarchyfile_path_template: Local file path used for CLI uploadshort_description: Description to appear on hovering over file name in file browserlong_description: Longer description to appear on file documentation pagefile_purpose: Assigns a tag to show up in a particular file-browser view configuration. Permissible values:Source view,Analysis view,Clinical view,MiscellaneousWithin the directory for each assay:The defined schema is used to structure information about pipeline-related files in the respective< assay >_output_API.json.Information related to YAML configurations (which are generated by the CIDC and configured with CIMAC IDs to run the pipelines), are described in the respective< assay > config.schema.json.Documentation related to each pipeline is in the respective< assay > .md.Developer SetupInstall necessary dependencies.pipinstall-rrequirements.dev.txtInstall and configure pre-commit hooks.pre-commitinstall
Package: nci-cidc-schemas
cidc-schemasThis repository contains formal definitions of the CIDC metadata model usingjson-schemasyntax and vocabulary.View documentation athttps://nci-cidc.github.io/cidc-schemas/InstallationTo install the latest released version, run:pipinstallnci-cidc-schemasDevelopmentProject Structurecidc_schemas/- a python module for generating, validating, and reading manifest and assay templates.schemas/- json specifications defining the CIDC metadata model.templates/- schemas for generating and validating manifest and assay templates.assays/- schemas for defining assay data models.artifacts/- schemas for defining artifacts.docs/- the most recent build of the data model documentation, along with templates and scripts for re-generating the documentation.template_examples/- example populated Excel files for template specifications inschemas/templates, and.csvs auto-generated from those.xlsxs that allow to transparently keep track of changes in them.tests/- tests for thecidc_schemasmodule..githooks/- git hooks, e.g. for auto-generating.csvs intemplate_examples/and.htmldocumentation files.Developer SetupInstall necessary dependencies.pipinstall-rrequirements.dev.txtInstall and configure pre-commit hooks.pre-commitinstallRunning testsThis repository has unit tests in thetestsfolder. After installing dependencies
the tests can be run via the commandpytesttestsBuilding documentationPre-commit hooks ensure documentation is automatically up-to date. To build the documentation manually, run the following commands:pythonsetup.pyinstall# install helpers from the cidc_schemas librarypythondocs/generate_docs.pyThis will output the generated html documents indocs/docs. If the updated docs are pushed up and merged into master, they will be viewable athttps://nci-cidc.github.io/cidc-schemas/.Using the Command-Line InterfaceThis project comes with a command-line interface for validating schemas and generating/validating assay and manifest templates.Install the CLIClone the repository and cd into itgitclonegit@github.com:NCI-CIDC/cidc-schemas.gitcdcidc-schemasInstall thecidc_schemaspackage (this adds thecidc_schemasCLI to your console)pythonsetup.pyinstallRuncidc_schemas --helpto see available options.If you're making changes to the module and want those changes to be reflected in the CLI without reinstalling thecidc_schemasmodule every time, runpython3-mcidc_schemas.cli[args]Generate templatesCreate a template for a given template configuration.cidc_schemasgenerate_template-mtemplates/manifests/pbmc_template.json-opbmc.xlsxValidate filled-out templatesCheck that a populated template file is valid with respect to a template specification.cidc_schemasvalidate_template-mtemplates/manifests/pbmc_template.json-xtemplate_examples/pbmc_template.xlsxValidate JSON schemasCheck that a JSON schema conforms to the JSON Schema specifications.cidc_schemasvalidate_schema-fshipping_core.json
Package: nci-ipynb
nci_ipynbThis work was inspired fromipynbname.When run in a Jupyter notebook, simply returns the notebook filename or the full path to the notebook.
I created this to help with automating posting blog posts written in Jupyter notebooks directly to
GitHub Pages.You would think there was already some built-in way to access the current notebook name, but it took many hours
of searching for a way to do it. As it seems many others did, I tried using Javascript, but the async nature of
JS meant that it was unreliable. Finally I stumbled on thispost.
I have refactored the code there so a user can get either the name or path, but credit for most of the code
goes to the author of this post, thanks!ExamplesGet the notebook name:importnci_ipynbnb_fname=nci_ipynb.name()Get the full path to the notebook:importnci_ipynbnb_path=nci_ipynb.path()Get the root directory to the notebook:importnci_ipynbnb_path=nci_ipynb.dir()LimitationsNote that this only reliably works when running a notebook in a browser. So it does not currently work for things like nbconvert or papermill.For VS Code there is aworkaround.
Package: ncl
Copyright (c) 2020 Michael Alves Bittencourt de Mello (mchl.bittencourt@gmail.com)Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.Description: readme
Platform: UNKNOWN
Description-Content-Type: text/x-rst
Package: nclib
nclib is a python socket library that wants to be your friend.nclib provides:Easy-to-use interfaces for connecting to and listening on TCP and UDP socketsThe ability to handle any python stream-like object with a single interfaceA better socket class, the Netcat objectConvenient receive methods for common socket usage patternsHighly customizable loggingInteractive mode, connecting the socket to your stdin/stdoutIntelligent detection of socket closes and connection dropsLong-running functions cleanly abortable with ctrl-cLots of aliases in case you forget the right method nameMechanisms to launch processes with their in/out streams connected to socketsLaunch a process with gdb attachedTCP and UDP server classes for writing simple python daemonsA script to easily daemonize command-line programsDocumentation is available athttps://nclib.readthedocs.io/and source code is
available athttps://github.com/rhelmot/nclibIf you are familiar with pwntools, nclib provides much of the functionaly that
pwntools’ socket wrappers do, but with the bonus feature of not being pwntools.
Package: nclpy
nclpyA Python package for working with NCLFree software: MIT licenseDocumentation:https://HJRubin.github.io/nclpyInstallationCondaconda install nclpy -c conda-forgeFeaturesAllows you to reproduce NCL-style graphics with Python commandsNot really sure if this is feasible but it would be niceCreditsThis package was created withCookiecutterand thegiswqs/pypackageproject template.
Package: ncls
Nested containment listThe Nested Containment List is a datastructure for interval overlap queries,
like the interval tree. It is usually an order of magnitude faster than the
interval tree both for building and query lookups.The implementation here is a revived version of the one used in the now defunct
PyGr library, which died of bitrot. I have made it less memory-consuming and
created wrapper functions which allows batch-querying the NCLS for further speed
gains.It was implemented to be the cornerstone of the PyRanges project, but I have made
it available to the Python community as a stand-alone library. Enjoy.Original Paper:https://academic.oup.com/bioinformatics/article/23/11/1386/199545Cite:http://dx.doi.org/10.1093/bioinformatics/btz615CiteIf you use this library in published research citehttp://dx.doi.org/10.1093/bioinformatics/btz615Installpip install nclsUsagefromnclsimportNCLSimportpandasaspdstarts=pd.Series(range(0,5))ends=starts+100ids=startssubject_df=pd.DataFrame({"Start":starts,"End":ends},index=ids)print(subject_df)#    Start  End# 0      0  100# 1      1  101# 2      2  102# 3      3  103# 4      4  104ncls=NCLS(starts.values,ends.values,ids.values)# python API, slowerit=ncls.find_overlap(0,2)foriinit:print(i)# (0, 100, 0)# (1, 101, 1)starts_query=pd.Series([1,3])ends_query=pd.Series([52,14])indexes_query=pd.Series([10000,100])query_df=pd.DataFrame({"Start":starts_query.values,"End":ends_query.values},index=indexes_query.values)query_df#        Start  End# 10000      1   52# 100        3   14# everything done in C/Cython; fasterl_idxs,r_idxs=ncls.all_overlaps_both(starts_query.values,ends_query.values,indexes_query.values)l_idxs,r_idxs# (array([10000, 10000, 10000, 10000, 10000,   100,   100,   100,   100,#          100]), array([0, 1, 2, 3, 4, 0, 1, 2, 3, 4]))print(query_df.loc[l_idxs])#        Start  End# 10000      1   52# 10000      1   52# 10000      1   52# 10000      1   52# 10000      1   52# 100        3   14# 100        3   14# 100        3   14# 100        3   14# 100        3   14print(subject_df.loc[r_idxs])#    Start  End# 0      0  100# 1      1  101# 2      2  102# 3      3  103# 4      4  104# 0      0  100# 1      1  101# 2      2  102# 3      3  103# 4      4  104# return intervals in python (slow/mem-consuming)intervals=ncls.intervals()intervals# [(0, 100, 0), (1, 101, 1), (2, 102, 2), (3, 103, 3), (4, 104, 4)]There is also an experimental floating point version of the NCLS called FNCLS.
See the examples folder.BenchmarkTest file of 100 million intervals (created by subsetting gencode gtf with replacement):LibraryFunctionTime (s)Memory (GB)bx-pythonbuild161.72.5nclsbuild3.150.5bx-pythonoverlap148.44.3nclsoverlap7.20.5Building is 50 times faster and overlap queries are 20 times faster. Memory
usage is one fifth and one ninth.Original paperAlexander V. Alekseyenko, Christopher J. Lee; Nested Containment List (NCList): a new algorithm for accelerating interval query of genome alignment and interval databases, Bioinformatics, Volume 23, Issue 11, 1 June 2007, Pages 1386–1393,https://doi.org/10.1093/bioinformatics/btl647
Package: nclustenv
NclustEnvNclustEnv is a python toolkit that implements several environments for n-clustering and other functionalities,
along with some default datasets and configurations.The motivation behindNclustEnvis providing quality general environments for training and benchmarking RL-based
solutions to n-clustering.NclustEnvwas implemented usingNclustGenas the generator for
online training, andGym.NclustEnv'sarchitecture focuses on an abstract class (base) that contains the core logic and API and is, consequently,
the only part that cannot be customized.The state abstraction:NclustEnvseparates all environment logic into a separate classState; this class can
be inherited for minor modification or re-implemented for significant changes. It receives actions from the main
class and actuates them in the graph. It is also is responsible for sampling the space and requesting a new episode
upon a reset command. Currently, there are two implemented states, theStateand theOffline Stateclasses.
Intuitively theStateclass handles all online environments, while theOffline Stateclass is used in offline
environments.The space abstraction:NclustEnvalso implements its observation spaceDGLHeteroGraphSpace, which can sample
graph configurations from a distribution used withNclustGenfor dataset creation.The metric abstraction:Thebaseclass implements the linear assignment function and all other core logic
necessary to estimate the reward and send it to the agent. However, it takes a function as a parameter so that other
reward functions might be used. This function should return the distance between all permutations of hidden and found
clusters. The only assumption made about the metric is that it is a distance metric; hence, the objective is to
minimize it.NclustEnvcurrently implementsJaccard Distance.The action abstraction:This abstraction implements a simple action container. When an action reaches the
environment is parsed through theActionclass. This class should implement two properties:actionthat contains the discrete action to take andparameterscontaining the vector of parameters for that action
index. Thebaseclass takes a pointer to this action class as a parameter. Nonetheless, it is not advised that
this action is re-implemented. Instead, to modify its behaviour, it should be inherited.InstallationThis tool can be installed from PyPI:pipinstallnclustenvGetting startedHere are the basics, for more information check the docstrings in each class.## Use Biclustering Environmentimportnclustenv# Initialize Environmentconfigs={}env=nclustenv.make('BiclusterEnv-v0',**configs)# Get obs spaceenv.observation_space# Get action spaceenv.action_space# Get state classstate=env.state# Step in environmentobs,reward,done,info=env.step(env.action_space.sample())# Render environmentenv.render()Currently four environments are implemented:'BiclusterEnv-v0';'TriclusterEnv-v0';'OfflineBiclusterEnv-v0';'OfflineTriclusterEnv-v0;LicenseGPLv3
Package: ncluster
nclusterBy Yaroslav Bulatov, Andrew Shaw, Ben Mannhttps://github.com/cybertronai/nclusterNcluster provides Python API to do the following things:Allocate AWS machineUpload file to machineRun command on machineDownload file from machineIEimport ncluster
task = ncluster.make_task(instance_type='p2.xlarge')
task.upload('myscript.py')
task.run('python myscript.py > out')
task.download('out')Necessary AWS infrastructure is created on demand using defaults optimal for fast prototyping. IE, your machines are preconfigured for passwordless SSH, can access each other over all interfaces, and have a persistent file system mounted under /ncluster. Commands are executed in a remote tmux session so you can take over the environment at any time and continue from your terminal.InstallationInstall pip, tmux, Python 3.6 (see below), andwrite downyour AWS security keys, thenpip install -r https://raw.githubusercontent.com/yaroslavvb/ncluster/master/requirements.txt
pip install -U ncluster
export AWS_ACCESS_KEY_ID=AKIAIBATdf343
export AWS_SECRET_ACCESS_KEY=z7yKEP/RhO3Olk343aiP
export AWS_DEFAULT_REGION=us-east-1Command-line toolsncluster
ncluster ls
ncluster hosts
ncluster ls
ncluster ls <substring>
ncluster ssh # connects to latest instance
ncluster ssh <substring>  # connects to latest instance containing <substring>
ncluster ssh \'<exact match>\'
ncluster mosh <substring> 
ncluster kill <substring>    # terminates matching instances
ncluster kill \'<exact match>\'
ncluster stop <substring>    # stops matching instances
ncluster start <substring>   # starts matching stopped instances
ncluster nano       # starts a tiny instance
ncluster keys   # information on enabling SSH access for your team-members

ncluster ssh_    # like ssh but works on dumb terminals
ncluster ls     
ncluster cat <fn>
ncluster cmd "some command to run remotely on AWS"

ncluster efs   # gives EFS info such as the mount command

nsync -m gpubox
nsync -m gpubox -d transformer-xl

nsync -d {target directory} -m {machine name substring}

nsync -m gpubox # syncs . to ~ on gpubox
nsync -d transformer-xl -m 4gpubox  # syncs . to ~/transformer-xl on 4gpubox


{substring} selects the most recently launched instances whose name contains the substring. Empty string is a valid substring. Skipping -t will sync to ~ on remote machine. Sync seems to be 1 way (from local -> remote)DocsSome out-of-date docs with more infodocsExtraAn example of installing pip/tmux/python 3.6 on MacOSDownload Anaconda distribution followinghttps://conda.io/docs/user-guide/install/index.htmlInstall tmux through homebrew:https://brew.sh/, thenbrew install tmuxThenconda create -n new python=3.6 -y
conda activate newExtra Deps:brew install fswatch
Package: nclustgen
NclustGenNclustgen is a python tool to generate biclustering and triclustering datasets programmatically.It wraps two java packagesG-Bic, andG-Tric, that serve as backend generators. If you are interested on a GUI version
of this generator or on using this generator in a java environment check out those packages.This tool adds some functionalities to the original packages, for a more fluid interaction with python libraries, like:Conversion to numpy arraysConversion to sparse tensorsConversion tonetworkXordgln-partite graphsInstallationThis tool can be installed from PyPI:pipinstallnclustgenNOTICE: Nclustgen installs by default the dgl build with no cuda support, in case you want to use gpu you can override this
by installing the correct dgl build, more information at:https://www.dgl.ai/pages/start.html.Getting startedHere are the basics, the full documentation is available at:http://nclustgen.readthedocs.org.## Generate biclustering datasetfromnclustgenimportBiclusterGenerator# Initialize generatorgenerator=BiclusterGenerator(dstype='NUMERIC',patterns=[['CONSTANT','CONSTANT'],['CONSTANT','NONE']],bktype='UNIFORM',in_memory=True,silence=True)# Get parametersgenerator.get_params()# Generate datasetx,y=generator.generate(nrows=50,ncols=100,nclusters=3)# Build graphgraph=generator.to_graph(x,framework='dgl',device='cpu')# Save data filesgenerator.save(file_name='example',single_file=True)## Generate triclustering datasetfromnclustgenimportTriclusterGenerator# Initialize generatorgenerator=TriclusterGenerator(dstype='NUMERIC',patterns=[['CONSTANT','CONSTANT','CONSTANT'],['CONSTANT','NONE','NONE']],bktype='UNIFORM',in_memory=True,silence=True)# Get parametersgenerator.get_params()# Generate datasetx,y=generator.generate(nrows=50,ncols=100,ncontexts=10,nclusters=25)# Build graphgraph=generator.to_graph(x,framework='dgl',device='cpu')# Save data filesgenerator.save(file_name='example',single_file=True)LicenseGPLv3DocumentationThe documentation is available at:https://nclustgen.readthedocs.org.