Saravanan666/pypi-org huggingface dataset:
Package: data-complexity
data-complexityThe Data Complexity Measures in PythonInstall$pipinstalldata-complexityHow it worksMaximum Fisher's Discriminant Ratio (F1)fromdcmimportdcmfromsklearnimportdatasetsiris=datasets.load_iris()X=iris.datay=iris.targetindex,F1=dcm.F1(X,y)Fraction of Borderline Points (N1)fromdcmimportdcmfromsklearnimportdatasetsbc=datasets.load_breast_cancer(as_frame=True)X=bc.data.valuesy=bc.target.valuesN1=dcm.N1(X,y)Entropy of Class Proportions (C1) and Imbalance Ratio (C2)fromdcmimportdcmfromsklearnimportdatasetsbc=datasets.load_breast_cancer(as_frame=True)X=bc.data.valuesy=bc.target.valuesC1,C2=dcm.C12(X,y)Other MeasuresComing soon...References[1] How Complex is your classification problem? A survey on measuring classification complexity,https://arxiv.org/abs/1808.03591[2] The Extended Complexity Library (ECoL),https://github.com/lpfgarcia/ECoL
Package: datacompy
DataComPyDataComPy is a package to compare two Pandas DataFrames. Originally started to
be something of a replacement for SAS'sPROC COMPAREfor Pandas DataFrames
with some more functionality than justPandas.DataFrame.equals(Pandas.DataFrame)(in that it prints out some stats, and lets you tweak how accurate matches have to be).
Then extended to carry that functionality over to Spark Dataframes.Quick InstallationpipinstalldatacompyorcondainstalldatacompyInstalling extrasIf you would like to use Spark or any other backends please make sure you install via extras:pipinstalldatacompy[spark]pipinstalldatacompy[dask]pipinstalldatacompy[duckdb]pipinstalldatacompy[polars]pipinstalldatacompy[ray]In-scope Spark versionsDifferent versions of Spark play nicely with only certain versions of Python below is a matrix of what we test withSpark 3.1.3Spark 3.2.3Spark 3.3.4Spark 3.4.2Spark 3.5.0Python 3.8✅✅✅✅✅Python 3.9✅✅✅✅✅Python 3.10✅✅✅✅✅Python 3.11❌❌❌✅✅Python 3.12❌❌❌❌❌:::{note}
At the current time Python3.12is not supported by Spark and also Ray within Fugue.
:::Supported backendsPandas: (See documentation)Spark: (See documentation)Polars (Experimental): (See documentation)Fugue is a Python library that provides a unified interface for data processing on Pandas, DuckDB, Polars, Arrow,
Spark, Dask, Ray, and many other backends. DataComPy integrates with Fugue to provide a simple way to compare data
across these backends. Please note that Fugue will use the Pandas (Native) logic at its lowest level
(See documentation)ContributorsWe welcome and appreciate your contributions! Before we can accept any contributions, we ask that you please be sure to
sign theContributor License Agreement (CLA).This project adheres to theOpen Source Code of Conduct.
By participating, you are expected to honor this code.RoadmapRoadmap details can be foundhere
Package: dataconda
DataCondaDataConda (http://easy-breezy.xyz/DataConda/https://github.com/ploppy-pigeon/DataConda), developed by Vinayak Vikram, developer of EasyBreezy is a python library for data.Its features include:
-   Tkinter Grahing widgets
-   Chart widgets (class Chart)
-   Graph widgets (only work with DataConda Chart widgets)(class Graph)
-   Data Servers/Web apps (class DataServer)
-   Converting
-   ParsingCredits:
-   StackOverflow for teaching me how to embed matplotlib charts in Tk (used for the Chart and Graph classes) (I fiddled around a bit also) (Link here:https://stackoverflow.com/questions/46332192/displaying-matplotlib-inside-tkinter)
Package: dataconsistent
Thesisof Michael PilosovThe text below is merely a placeholder taken from the MUD repo.Analytical solutions and some associated utility functions for computing maximal updated density points for Data-Consistent Inversion.DescriptionMaximal Updated Density Points are the values which maximize an updated density, analogous to how a MAP (Maximum A-Posteriori) point maximizes a posterior density from Bayesian inversion.
Updated densities differ from posteriors in that they are the solution to a different problem which seeks to match the push-forward of the updated density to a specified observed distribution.More about the differences here...What does this package include?NoteThis project has been set up using PyScaffold 3.2.3. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.
Package: data-conversion
Data-Conversion is a framework to convert data from origin style to target style easily.
With custom settings, data-conversion can read data from MongoDB, convert
data by MAPPING Rules in settings, and save to destination collection in MongoDB.How to InstallInstall by pip:$ pip install data-conversionHow to UseFirst, you should create a new settings file, for example,settings_release.py.
Then, define custom settings like Setting Template Filesettings.pyindata_conversion/settings.py, whose arguments also describe below.
Finally, run asynchronously:$ etl async settings_release.pyor run synchronously:$ etl sync settings_release.pySettingsArgumentDescriptionValue ExampleMONGODB_HOSTHost of MongoDB which store origin data‘127.0.0.1’MONGODB_PORTPort of MongoDB which store origin data27017MONGODB_USERNAMEUsername of MongoDB which store origin dataNone / ‘admin’MONGODB_PASSWORDPassword of MongoDB which store origin dataNone / ‘123456’MONGODB_AUTHDBDB of authorization which store username and password‘admin’MONGODB_DBDB of MongoDB which store origin data and will store result data‘data’MONGODB_SRC_COLLSource Collection of MongoDB which store origin data‘src_coll’MONGODB_DST_COLLDestination Collection of MongoDB which will store result data‘dst_coll’MONGODB_DST_COLL_INDEXDestination Collection Index of MongoDB which store result data[([(‘url’, pymongo.ASCENDING)], {‘unique’:True}), ([(‘domain’, pymongo.ASCENDING)], {})]MONGODB_ERROR_COLLError Collection of MongoDB which will store error data when convert raise exception‘error_coll’MONGODB_ERROR_COLL_INDEXCollection Index of Error Collection of MongoDB[([(‘url’, pymongo.ASCENDING)], {‘unique’: True})]SRC_COLL_QUERYQuery condition to select documents to be converted{ ‘filter’: {}, ‘projection’: None, ‘start’: 0, ‘limit’: 1000 }WRITE_CONDITION_DICTwrite to dst_coll which collection.update({CONDITION}, {$set:{dst_document}}, upsert=True){‘$set’: [‘url’]}MAPPINGlist to mapper, rules of conversion[Mapper(‘url’, ‘url’, str, None)] // src_key, dst_key, dst_type, custom_convert_functionOPERATE_MAPPING_DICTdict to mapper, rules of conversion{‘$set’:MAPPING, ‘$push’: MAPPING2, ‘$addToSet’: MAPPING3}PROCESS_NUMNumber of process to run conversion1CONCURRENT_PER_PROCESSnumber of concurrent group to run in one process100LOG_LEVELLevel of logginglogging.INFOSettings explainThe most important part in settings is MAPPING. MAPPING contains a list of Mapper,
which is a namedtuple (src_key, dst_key, dst_type, custom_convert).
dst_type and custom_convert can beNoneif you want to preserve origin type and value.Now, we support ‘$set’, ‘$push’, ‘$addToSet’ operation when update document,
if you want to add each array element to an existed array,
please add ‘$each_’ by custom_convert_function.
.. _$eachhttps://docs.mongodb.com/manual/reference/operator/update/addToSet/#each-modifierException HandlingException occured in convert function will be save into error collection which
defined in settings.If you want to record the key of document which excpetion raise, you canraiseValueError('key')containskeyas an argument.
Package: data-converter
Convert data files between different formatsInstallPython:pip installdata-converterBrew:comingsoon...Ways to useCommand line$ data-converter -h
usage: data-converter [-h] -t TO input_file

Convert data files

positional arguments:
  input_file      File to convert from

optional arguments:
  -h, --help      show this help message and exit
  -t TO, --to TO  Format to convert toImport in pythonimportdata_converter# Convert csv to json fileoutput_path=data_converter.convert("~/some/data/file.csv","json")Currently supported file typesjsoncsv
Package: dataconvertor
Р­С‚Р° Р±РёР±Р»РёРѕС‚РµРєР° РґР»СЏ РєРѕРЅРІРµСЂС‚Р°С†РёРё json РІ С„РѕСЂРјР°С‚ [РєР»СЋС‡Рё],[Р·РЅР°С‡РµРЅРёСЏ]
Package: datacoral-tracker-py
Datacoral's Python Instrumentation using SnowplowPrerequisitsThis section of the documentation is useful if you have theDatacoral Events Connectoralready added to your Datacoral Installation.Contactsupport@datacoral.cotoday, in oredr to set up your new Events Connector.OverviewOur default Collect Events Slice is compatible with theSnowplow Tracker Protocol.Snowplow provides tracker or instrumentation libraries in several languages.We have incorporated those libraries and enhanced them to support:Pointing to your own Events HTTP Endpoint.Generate API KeysCreate multiple environments like dev/stage/prod so that events can be segregated based on the environment.A browser js instrumentation, specifyingCORS OriginsContributing quickstartAssumingGitandDatacoral CLIare installed:Installing datacoral-tracker-pypip install datacoral-tracker-pyPublishing (Tracking)fromsnowplow_trackerimportSelfDescribingJson,Tracker,Emitterapi_endpoint=os.environ['API_GATEWAY_ENDPOINT']datacoral_env=os.environ['DATACORAL_ENV']api_key=os.environ['DATACORAL_API_KEY']namespace=os.environ['NAMESPACE']or'mynamespace'app_id=os.environ['APP_ID']or'datacoral-python-tracker'# Initialize the emitteremitter=Emitter(api_endpoint,buffer_size=1)# Initialize the trackertracker=Tracker(emitter,datacoral_env=datacoral_env,api_key=api_key,namespace=namespace,app_id=app_id,encode_base64=True)# Send a custom event with your own schema and contextsdata={"foo":"bar","metadata":{"sub":"data"}}sjson=SelfDescribingJson(schema="test",data=data)tracker.track_unstruct_event(event_json=sjson)Additional ReadingEvents Slice OverviewEvent from PythonCopyright and licenseLicensed under theApache License, Version 2.0(the "License");
you may not use this software except in compliance with the License.Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: datacorral
datacorral: Python data toolkitWhat is it?Data Corralis a package that provides helper functions to assist in data extraction, loading, manipulation and analysis.Where to get itThe source code is currently hosted on GitHub at:https://github.com/sheplers/datacorralBinary installers for the latest released version are available at thePython
Package Index (PyPI)# or PyPIpipinstalldata_corralDependenciesPandas - Adds support for multidimensional arrays and high-level mathematical functionsDocumentationThe official documentation is hosted on
Package: datacraft-xeger
datacraft-xegerCustom plugin fordatacraftto generate values using regular expressions.
Uses therstrpackage.xegeris regex backwards. Inspiration from the
original Java Packagexeger.Usage in SpecsYou can use thexegeras a type in your datacraft data specs. See example:{"ssn":{"type":"xeger","data":"\\d{3]-\\d{2}-\\d{4}"}}$datacraft-sxeger.json-i3--formatjson-pretty-x-lerror[{"ssn":"322-81-1469"},{"ssn":"697-21-8178"},{"ssn":"340-78-5377"}]Custom Regex TypesUsers can make use of the datacraft_xeger module to create custom datacraft value suppliers with regex patterns. The
example below shows how to register custom types for different country phone number patterns.importdatacraftimportdatacraft_xeger.suppliersasxegerphone_patterns={# type_name: pattern'uk-phone':r'\+44 \d{4}\d{6}','aus-phone':r'\+61 4\d{2}\d{3}\d{3}','nz-phone':r'\+64 \d{2}\d{4}\d{4}',# ...}@datacraft.registry.types('uk-phone')def_custom_regex_uk_phone(spec,loader):returnxeger.xeger_supplier(phone_patterns['uk-phone'])@datacraft.registry.types('aus-phone')def_custom_regex_aus_phone(spec,loader):returnxeger.xeger_supplier(phone_patterns['aus-phone'])@datacraft.registry.types('nz-phone')def_custom_regex_nz_phone(spec,loader):returnxeger.xeger_supplier(phone_patterns['nz-phone'])Once registered these types can be used as part of the data generation process.  See the example data spec:{"name":["ann","bob","carl"],"age":{"type":"rand_int_range","data":[25,75]},"phone":{"type":"weighted_ref","data":{"UK":0.5,"AUS":0.3,"NZ":0.2}},"refs":{"UK":{"type":"uk-phone"},"AUS":{"type":"aus-phone"},"NZ":{"type":"nz-phone"}}}Running datacraft against this spec and using the custom code loading feature:datacraft-scustom.json-ccustom.py-i3--formatjson-pretty-x-lwarn[{"name":"ann","age":67,"phone":"+64 07 2500 7403"},{"name":"bob","age":49,"phone":"+61 435 126 947"},{"name":"carl","age":61,"phone":"+44 7693 148185"}]
Package: datacrunch
DataCrunch Python SDKThe officialDataCrunch.ioPython SDK.The SDK's documentation is available onReadTheDocsDataCrunch's Public API documentationis available here.Getting Started - Using the SDK:Install via pip:pip3installdatacrunchGenerate your client credentials -instructions in the public API docs.Add the client secret to an environment variable (don't want it to be hardcoded):Linux (bash):exportDATACRUNCH_CLIENT_SECRET=Z4CZq02rdwdB7ISV0k4Z2gtwAFKiyvr2U1l0KDIeYiOther platforms:https://en.wikipedia.org/wiki/Environment_variableExample for creating a new instance:importosfromdatacrunchimportDataCrunchClient# Get client secret from environment variableCLIENT_SECRET=os.environ['DATACRUNCH_CLIENT_SECRET']CLIENT_ID='Ibk5bdxV64lKAWOqYnvSi'# Create datcrunch clientdatacrunch=DataCrunchClient(CLIENT_ID,CLIENT_SECRET)# Get all SSH keysssh_keys=datacrunch.ssh_keys.get()ssh_keys=list(map(lambdakey:key.id,ssh_keys))# Create a new instanceinstance=datacrunch.instances.create(instance_type='1V100.6V',image='fastai',ssh_key_ids=ssh_keys,hostname='example',description='example instance')# Delete instancedatacrunch.instances.action(instance.id,datacrunch.constants.instance_actions.DELETE)More examples can be found in the/examplesfolder or in thedocumentation.DevelopmentSetting up the local development environmentClone the repository:gitcloneCreate local virtual environment:python3-mvenvdatacrunch_env&&source./datacrunch_env/bin/activateor if usingfish shell:python3 -m venv datacrunch_env&&source./datacrunch_env/bin/activate.fishInstall Dependencies:pip3install-e.[test]pip3install-UpytestRunning TestsWe use pytest for testing.To execute all testspytestTo execute a single test filepytest./tests/unit_tests/test_file.pyLocal Manual TestingCreate this file in the root directory of the project:fromdatacrunch.datacrunchimportDataCrunchClientCLIENT_SECRET='secret'CLIENT_ID='Ibk5bdxV64lKAWOqYnvSi'# Create datcrunch clientdatacrunch=DataCrunchClient(CLIENT_ID,CLIENT_SECRET,base_url='http://localhost:3001/v1')Generating the documentationIf added a new service, create a documentation template under api/services for that service.cddocs
makehtmlStyle GuideUse autopep8 for auto code formatting:# Installpip3installautopep8# Apply to an entire directoryautopep8directory_name--recursive--in-place--pep8-passes2000--verbose# Or a single fileautopep8file.py--in-placeContactYou cancontact us here, or send a message / open an issue in the repo.
Package: datacube
Open Data Cube CoreOverviewThe Open Data Cube Core provides an integrated gridded data
analysis environment for decades of analysis ready earth observation
satellite and related data from multiple satellite and other acquisition
systems.DocumentationSee theuser guidefor
installation and usage of the datacube, and for documentation of the API.Join our Slackif you need help
setting up or using the Open Data Cube.Please help us to keep the Open Data Cube community open and inclusive by
reading and following ourCode of Conduct.RequirementsSystemPostgreSQL 10+Python 3.9+Developer setupClone:git clonehttps://github.com/opendatacube/datacube-core.gitCreate a Python environment for using the ODC.  We recommendMambaforgeas the
easiest way to handle Python dependencies.mamba env create -f conda-environment.yml
conda activate cubeenvInstall a develop version of datacube-core.cd datacube-core
pip install --upgrade -e .Install thepre-commithooks to help follow ODC coding
conventions when committing with git.pre-commit installRun unit tests + PyLintInstall test dependencies using:pip install--upgrade-e'.[test]'If install for these fails, please lodge them as issues.Run unit tests with:./check-code.sh(this script approximates what is run by GitHub Actions. You can
alternatively runpytestyourself).(or)Run all tests, including integration tests../check-code.shintegration_testsAssumes a password-less Postgres database running on localhost calledpgintegrationOtherwise copyintegration_tests/integration.confto~/.datacube_integration.confand edit to customise.For instructions on setting up a password-less Postgres database, seethedeveloper setup instructions.Alternatively one can use theopendatacube/datacube-testsdocker image to run
tests. This docker includes database server pre-configured for running
integration tests. Add--with-dockercommand line option as a first argument
to./check-code.shscript../check-code.sh --with-docker integration_testsTo run individual tests in a docker containerdocker build --tag=opendatacube/datacube-tests-local --no-cache --progress plain -f docker/Dockerfile .

docker run -ti -v $(pwd):/code opendatacube/datacube-tests-local:latest pytest integration_tests/test_filename.py::test_function_nameDeveloper setup on UbuntuBuilding a Python virtual environment on Ubuntu suitable for development work.Install dependencies:sudo apt-get update
sudo apt-get install -y \
    autoconf automake build-essential make cmake \
    graphviz \
    python3-venv \
    python3-dev \
    libpq-dev \
    libyaml-dev \
    libnetcdf-dev \
    libudunits2-devBuild the python virtual environment:pyenv="${HOME}/.envs/odc"  # Change to suit your needs
mkdir -p "${pyenv}"
python3 -m venv "${pyenv}"
source "${pyenv}/bin/activate"
pip install -U pip wheel cython numpy
pip install -e '.[dev]'
pip install flake8 mypy pylint autoflake black
Package: datacubes
DataCubesA client library for InfoBlox DDI appliances.FeaturesTODOCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2020-02-24)First release on PyPI.0.1.1 (2020-02-24)Attempt to fix PyPI homepage URL.
Package: datacula
DATACULAUnveiling Datacula, the Dracula of Python packages, fearlessly devouring complexities in the analysis and data collection from aerosol-gas instruments. Developed by the creators ofParticula, Datacula offers seamless integration with their cutting-edge modeling tools.Datacula furnishes you with a comprehensive suite of tools to import, process, and visualize data from a diverse range of instruments in air quality monitoring. If your instrument data format isn't supported yet, you can contribute by adding it, thereby paving the way for future scientists.For those with instrumental data, discover Particula—your go-to aerosol particle simulator. Designed for simplicity and adaptability, Particula delivers a robust aerosol simulation system for both gas and particle phases. Harness its power to tackle scientific questions arising from your Datacula data.Goals & conductThe main goal is to develop an aerosol data management and processing tool that is usable, efficient, and productive. In this process, we all will learn developing models in Python and associated packages. Let us all be friendly, respectful, and nice to each other. Any code added to this repository is automatically owned by all. Please speak up if something (even if trivial) bothers you. Talking through things always helps. This is an open-source project, so feel free to contribute, however small or big your contribution may be.We follow the Google Python style guidehere. We have contribution guidelineshereand a code of conducthereas well.Usage & developmentThe development of this package will be illustrated through Jupyter notebooks (here) that will be put together in the form of a Jupyter book on our (to be added).For development, you can fork this repository and then installparticulain an editable (-e) mode --- this is achieved bypip install -e ".[dev]"in the root of this repository. Invokingpip install -e ".[dev]"will installparticula, its runtime requirements, and the development and test requirements. The editable mode is useful because it allows seeing the manifestation of code edits globally through theparticulapackage in your environment (in a way, with the-emode,particulaself-updates to account for the latest local code edits).Python versionWe support python 3.9 and above. To check your python version, runpython --versionin your terminal. To upgrade your python version, runpip install --upgrade pythonor with condaconda install python=3.10.Tour & ExamplesTBD
Package: datacustodian
[![pipeline status](https://gitlab.com/clearos/clearfoundation/datacustodian/badges/master/pipeline.svg)](https://gitlab.com/clearos/clearfoundation/datacustodian/commits/master) [![coverage report](https://gitlab.com/clearos/clearfoundation/datacustodian/badges/master/coverage.svg)](https://gitlab.com/clearos/clearfoundation/datacustodian/commits/master)#DataCustodianData Privacy Package GeneratorDataCustodiangenerates other repositories for generalized
data privacy and consent systems. The generation follows YML specification files
that define the endpoints, handlers, authentication, validation, etc. for each
component of a data privacy solution.DataCustodianis built to work with decentralized and distributed systems to
provide decentralized trust for storage and consent of data.Check out the [API Documentation](https://clearos.gitlab.io/clearfoundation/datacustodian) and a [real-world implementation](https://gitlab.com/clearos/clearfoundation/clearshare).## InstallationThe package is available from the PyPI repository:`bash pip install datacustodian `## Generating a PackageLook at the documentation indocs/configsfor an example setup that we use for
the unit tests. Customize that to your own application and then run:`bash datacustodian_app.py path/to/your/app_spec.yml--generate`That will auto-generate the packageandstart the REST API server.## Running Unit TestsRunning unit tests can prove to be a massive pain because the gitlab CI runner
runs on docker. For a project like this that usesdocker-compose, we have
to usedocker-in-dockeraccording to their instructions. However, the
documentation is sparse and there are lots of dead-ends… Here are the steps to
get thedocker-compose.ymlfile to work:Install a localgitlab-runnerusingbrew install gitlab-runner.gitlab-runner exec docker –docker-privileged test. Notice that there is a–docker-privilegedargument. Without that, thedocker-in-dockerwon’t
work.Make sure all themultiaddrreference thedockerservice (which hosts
all the other containers usingdind).toxshould work, but for some reason: running the tests usingtoxproduces
connection refused errors, whereas running straight withpytestdoes not.
Something about the tox environment screws things up.## Problems withcoverageFor an yet-unknown reason, when the unit tests run in the CI server using:Onlypytest, they pass with no issues.coverage runwithpytest, we get problems with connection reset errors,
and connection refused errors.Insead of trying to figure this out now, we just do the following:Before pushing to the remote server, runcoveragelocally (which has no
problems on MacOS).Generate thecoverage report -m > codecov.out.Commitcodecov.outand then push.The CI serveronlyrunspytest, but it alsocat codecov.outso that the
output includes the regular code coverage matrix. That way,gitlabstill has
correct statistics about code coverage.