Saravanan666/pypi-org huggingface dataset:
Package: datasimple
Author:Craig KellyIntroductionSource Note:The authoritative version of this file is the Markdown
version. The RST version is automatically created from the Markdown bypandoc.This is an Apache licensed library and set of command-line tools for
simple data processing tasks and pipelines. It is assumed that it will
be used with tools likedmkand
that serious work will be done with serious tools (like jupyterlab and
scipy).If it feels like a mishmash of functionality, that’s because it is. This
is mainly a collection of odds and ends that keeps getting used in
projects in a very specific analytics and data science team.InstallingThe normal way:$ pip install datasimpleHowever, we use Python 3 and prefer user installs, so on a system like
Ubuntu you probably want:$ python3 -m pip install --user --upgrade datasimpleHOWEVER, The CORRECT usage is a Pipfile controlled by pipenv.See below (inHacking) for installing in development mode
if you need to make source code changes.What you getThe datasimple library and some handy scripts (see./bin). Of note
is a class designed to help you write scripts to convert anything to
Excel spreadsheets. (Once again, this is functionality we need for a
particular business environment. It is expressly NOT an endorsement of
Excel for data science.)RequirementsThis is Python 3. Don’t submit requests for Python 2 compatibility.See setup.py for dependencies (which will get installed automatically
when you install this package withpip)HackingYou should be developing in a virtualenv. Since you are probably forced
to work in a Vagrant Ubuntu VM on a Windows machine, and you’ll want to
use the shared/vagrantfolder, you might want to consider using
pipenv and pyenv with the virtualenv plugin.Usemake testfor testing (which will also handle linting). In fact,
see theMakefilefor what we automate with this project.ContributingThe following guidelines are used when accepting external contributions:./lintshould not find any issuesThere should be appropriate tests add to the appropriate module in./testsThere should be an existingandcompelling use case.The./lintscript in the root of this repo uses pylama which you
must install. Currently it also expects a pylama linter plugin called
“quotes”. See Craig (the maintainer) for this plugin. NOTE: if even ONE
PERSON contacts me I’ll make that plugin public :)If you don’t currently have pylama installed, you can get the latest
installed for your user withpython3-mpip install--user--upgradepylama.You should also test using the./testscript in the root of this
repo. It runs tests using nosetests. Our setup also requires the package
nose-exclude. However, the test script delegates viasetup.pyso you
shouldn’t need to worry about this.Note that both pylama and nosetests have configuration specified insetup.cfg.
Package: datasiphon
DatasiphonPackage for applying dictionary filter to some form of query on database to retrieve filtered data or acquire filtered queryInstallationUse the package managerpipto install datasiphon.pipinstalldatasiphonUsagefromsiphonimportbuild,sqlimportsqlalchemyassa# Create a filterfilter_={"name":{"eq":"John"},}table=sa.Table("users",sa.MetaData(),autoload=True,autoload_with=engine)# Build a queryquery=table.select()# apply filter using build functionquery=build(query,sql.SQl,filter_)# execute queryresult=engine.execute(query)...Currently Supported DatabasesSQL (using sqlalchemy)
Package: datasist
Getting Starteddatasist: Python library for easy data analysis, visualization, exploration and modelingLatest ReleaseRelease StatusLicenseBuild StatusDownloadsSource RankTwitterWhat is it?datasistis a python package providing fast, quick, and an abstracted interface to popular and frequently used functions or techniques relating to data analysis, visualization, data exploration, feature engineering, Computer, NLP, Deep Learning, modeling, model deployment etc.InstallpipinstalldatasistInstallation from source (Developers)To install datasist from source you need python 3.6> in addition to the normal dependencies above.Run the following command in a terminal/command promptgitclonehttps://github.com/risenW/datasist.gitcddatasist
pythonsetup.pyinstallAlternatively, you can use install withpipafter cloning, if you want all the dependencies pulled in automatically (the-eoption is for installing it in [development mode]:gitclonehttps://github.com/risenW/datasist.gitcddatasist
pipinstall-e.DocumentationFull documentaion can be foundhereContributing to datasistAll contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.If you would like to contribute to datasist, find a detailed contribution guidehere.Using DatasistQuick start guideTutorial guideYoutube VideosIntroduction to using Datasist, presented at NeuRips Meetup - PH.Click HereIntroduction to Datasist by DataKnightClick Here
Package: datasize
DataSizePython integer subclass to handle arithmetic and formatting of integers with data size unitsProvides parsing, arithmetic and comparison oprations, and formatting of human readable data size strings for logic that depends on comparisons of values given in common units of data allocation. There are other solutions, but they are either not complete, or too heavy or awkward for casual use. A string like "14GiB" is really an integer representing a data allocation.The basic use case is to be able to parse a string containing a common expression of data size with a numeric value and a unit of data. The resulting object is actually an integer count of bytes, so that it can be used in any arithmetic expression. That integer can be expressed, using Python 3 string formatting, as any other unit of data. This allows, for example, configuration files that support a natural way of expressing and operating on quantities of data.DataSize supports metric and IEC units in both bits and bytes and nonstandard abbreviated IEC units (for legacy Java -Xmx). There is support for variable word-lengths, but because I thought it would get confusing, converting between two different word lengths is not supported. The word length constructor keyword argument will allow converting counts of weird (actually non-byte) word or symbol bit lengths to bit rates, which can then be explicitly converted to standard 8-bit bytes.The really sweet feature that everyone should love is the Python string.format() support!Help on method __format__ in module datasize.DataSize:

__format__(self, code) unbound datasize.__datasize__.DataSize method
    formats as a decimal number, but recognizes data units as type format codes.
    Precision is ignored for integer multiples of the unit specified in the format code.
    format codes:  
    a    autoformat will choose a unit defaulting to the largest
          size with a quantity >= 1 (default)
    A    abbreviated number of bytes (implied IEC units, and implied 'B' bytes suffix omitted)
    B    bytes      (1)
    kiB  kibibytes  (1024)
    kB   kilobytes  (1000)
    ...
    GiB  Gibibytes  (1024**3)
    GB   Gigabytes  (10**9)
    ...
    YiB  Yobibytes  (1024**8)
    YB   Yottabytes (10**24)>>> from datasize import DataSize
>>> 'My new {:GB} SSD really only stores {:.2GiB} of data.'.format(DataSize('750GB'),DataSize(DataSize('750GB') * 0.8))
'My new 750GB SSD really only stores 558.79GiB of data.'
Package: datasketch
datasketch gives you probabilistic data structures that can process and
search very large amount of data super fast, with little loss of
accuracy.This package contains the following data sketches:Data SketchUsageMinHashestimate Jaccard similarity and cardinalityWeighted MinHashestimate weighted Jaccard similarityHyperLogLogestimate cardinalityHyperLogLog++estimate cardinalityThe following indexes for data sketches are provided to support
sub-linear query time:IndexFor Data SketchSupported Query TypeMinHash LSHMinHash, Weighted MinHashJaccard ThresholdMinHash LSH ForestMinHash, Weighted MinHashJaccard Top-KMinHash LSH EnsembleMinHashContainment ThresholdHNSWAnyCustom Metric Top-Kdatasketch must be used with Python 3.7 or above, NumPy 1.11 or above, and Scipy.Note thatMinHash LSHandMinHash LSH Ensemblealso support Redis and Cassandra
storage layer (seeMinHash LSH at Scale).InstallTo install datasketch usingpip:pip install datasketchThis will also install NumPy as dependency.To install with Redis dependency:pip install datasketch[redis]To install with Cassandra dependency:pip install datasketch[cassandra]
Package: datasloth
Natural language Pandas queries and data generation powered by GPT-3Installationpip install dataslothUsageIn order for DataSloth to work, you must have a workingOpenAI API
keyset in your
environment variable, or provide it to the DataSloth object. For more
info, refer to thisguide.DataSloth automatically discovers all Pandas dataframes in your
namespace (filtering out names starting with an underscode). Before you
load any data, import DataSloth and create thesloth:fromdataslothimportDataSlothsloth=DataSloth()Next, load any data you want to use. Try naming your dataframes and
columns in a meaningful way, as DataSloth uses these names to understand
what the data is about.Once your data is loaded, simply runsloth.query('...')to query the data.
Package: dataslots
dataslotsDecorator for addingslotsIn python 3.7 dataclasses module was introduced for faster class creation (PEP 557).
Unfortunately, there's no support for__slots__(basic support was added in 3.10). If you want to create more memory
efficient instances, you need to do it by yourself or use@dataslotsdecorator.UsageSimple example@dataslots@dataclassclassPoint2D:x:inty:intInheritanceAs described in docs, in derived class__dict__is created, because base class does not have__slots__.
Slots are created from all defined properties (returned bydataclasses.fields()function).@dataclassclassBase:a:int@dataslots@dataclassclassDerived(Base):c:intd:intDynamic assignment of new variables@dataslots(add_dict=True)@dataclassclassPoint2D:x:inty:intpoint=Point2D(10,20)point.length=math.sqrt(point.x**2+point.y**2)Weakref@dataslots(add_weakref=True)@dataclassclassPoint2D:x:inty:intpoint=Point2D(10,20)r=weakref.ref(point)Read-only class variablesWith__slots__it's possible to define read-only class variables. When using dataclasses you cannot provide type
for attribute or usetyping.ClassVarto declare one.@dataslots@dataclassclassA:x=5y:ClassVar[set]=set()Pickling frozen dataclassBecause of anissue 36424you need custom__setstate__method. In dataslots
there is implemented default version, and it is used if decorated class has no__getstate__and__setstate__function declared.Added in 1.0.2Data descriptorsData descriptorsare supported by
inheritance fromDataDescriptor(base class with required interface) orDataslotsDescriptor(class with
additional features to simplify descriptor definition).Check example directory for basic usage.Added in 1.1.0Typing support (PEP 561)The package is PEP 561 compliant, so you can easily use it with mypy1and pyright.1Due to some issues in mypy not all features are supported correctly (likedataclass alike
interfaceordescriptors).Added in 1.2.0BackportIf you prefer using the newestdataclasses.dataclassinterface you can usedataslots.dataclasswrapper
to provide a consistent interface regardless of the python version.Notice: Wrapper always usesdataslotsto make all additional features available andslots=Trueis obligatory.Added in 1.2.0SLSA supportAll packages from version 1.2.0 can be verified usingSLSA provenance(dataslots package is compliant withSLSA Level 3).If you want to verify dataslots before installing, you need to downloadSLSA verifierand run:slsa-verifierverify-artifact\--provenance-pathdataslots.intoto.jsonl\--source-urigithub.com/starhel/dataslots\--source-tagv${VER}\${PATH_TO_PACKAGE}VERis version of package download from PYPI or GH release. Provenance is only available in GH release as PYPI
does not accept jsonl files.More about __slots__https://docs.python.org/3/reference/datamodel.html#slotshttps://github.com/ericvsmith/dataclasses/issues/28
Package: data-snack-dynamic-entity
Data Snack - Dynamic entityAboutUsed to dynamically loaddata-snackEntityobjects from json schema.
Especially useful for complex dataset with many fields.InstallData Snack can be easily installed using pypi repository.pipinstalldata_snack_dynamic_entityUsage1. Define Entity templateFirst you need to define a dictionary containing the template - a configuration for your Entity type.
Template should contain:the name of the new Entity type and its propertieseach property should have:type- type of the fielddefault- default value (optional property)optional- true if the field is optional (optional property)Notice that your template can be saved in any file of your choosing.
Just make sure it's later parsed to a dictionary in the right format.The template is defined using following schema:src/data_snack_dynamic_entity/entityTemplates.schema.json.Example{"Car":{"properties":{"name":{"type":"str"},"cost":{"type":"float","default":10.0,"optional":True}}}}2. Load the template and create new typesNow you are ready to create entities based on your template:fromdata_snack_dynamic_entity.factoryimportload_entitiesentities=load_entities(templates=your_config)After thatentitieswill contain a dictionary in the following format:{"entity_name":EntityType}ContactPlugin was created by the Data Science team fromWebinterpret.
Package: datasoap
datasoapWhat is it?datasoap is a supplementary library for pandas that processes dataframes derived from CSV files. The module checks cell data for correct numerical formatting and converts mismatched data to the correct data type (ex. str > float64).Main FeaturesStrips unnecessary characters from numerical data fields in pandas dataframes to ensure consistent data formattingProvides before and after representations of dataframes to allow for comparisonRepositorySource code is hosted on:github.com/snake-fingers/data-soapDependenciespandas - Python package that provides fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive.Installationpoetry add datasoapDocumentationUser Docs can be found atdatasoap docsBackgrounddatasoap originated from a Code Fellows 401 Python midterm project. The project team includes Alex Angelico, Grace Choi, Robert Carter, Mason Fryberger, and Jae Choi. After working with a few painful datasets using, we wanted to create a library that allows users to more efficiently manipulate clean datasets extracted from CSVs that may have inconsistent formatting.Future Features in PlanningAdditional functionality that re-names columns with the common unit of measure.Automation of identifying columns that require reformatting, removing the need for user input on instantiationExtend useablity to work with other virtual environments such as pip, and conda
Package: datasol
DATASOL-packageThis package is made for Seagate.
The package allows the user to detect anomalies in the data.To Install the package:pip install datasolCreating_Dataframe function will clean the data from unneeded columns, can transfer categorical to numeric, fill NAs with knn imputer, and save the file afterward.input- data frame (pandas), columns that you want to keep, and date column the rest are optional
By setting the next options True:
categorical_to_numeric is categorical columns to numeric.fillna is filling the Na values with knn imputer.save is saving the file to CSV.output- None if save put on True else Data FrameLoad_Dataframe function will read the CSV file and return the data frame after cleaning.input â€“ file directory, columns that you want to keep, and date column.output- Data FrameAnomaly_Detetion function will create a list for each column with anomaly data.input - data frameworks better if you use the output data frame from previous functions, time_step should be calculated by the user example:
Prepare training data
Get data values from the training time series data file and normalize the value data. We have a value for every 5 mins for 14 days.24 * 60 / 5 = 288 timesteps per day
288 * 14 = 4032 data points in total*The right amount of time steps will provide better results for anomaly detection. *model settings- epochs, batch_size, patience, layer1-4epochs -An epoch is when all the training data is used at once and is defined as the total number of iterations of all the training data in one cycle for training the machine learning model.batch_size â€“ the number of samples we want to pass into the training loop at each iteration.patience -After many tries, the model will have to improve before early stopping.layers 1-4 - in the base model there is 4 layer you can change how many filters each layer will have to improve performance.The threshold is set to 99% in other words the 1% data that is problematic and has a high chance to be an anomaly.visual if set to true will return a plot of the data and the anomaly.the function will return the anomaly list.Original data frame:After Creating_Dataframeoutput of Anomaly_Detetion
Package: datasouls-antispoof
Anti spoofing with the Datasouls datasetInstallationpip install -U datasouls_antispoofExample inferenceColab notebook with the example:DatasetID & RD anti spoofing challengeFour types of images:realreplayprintedmask2dTrainingDefine the config.Example atdatasoluls_antispoof/configsDefine the environmental variableIMAGE_PATHthat points to the folder with the dataset.Example:exportIMAGE_PATH=<pathtothefolderwithimages>Inferencepython-mtorch.distributed.launch--nproc_per_node=<num_gpu>datasouls_antispoof/inference.py\-i<pathtoimages>\-c<pathtoconfig>\-w<pathtoweights>\-o<output-path>\--fp16Pre-trained modelsModelsValidation accuracyConfig fileWeightsswsl_resnext50_32x4d0.9673LinkLinktf_efficientnet_b3_ns0.9927LinkLink
Package: datasoup
importdatasoupasdsdsdb=ds.db.connect("<connection_string>",options={})dsdb.project.list()dsdb.project.set("<project_name>",in_place=True)ds_project=dsdb.project.set("<project_name>",in_place=False)dsdb.project["<project_name>"].some_method()thoughtsuse symlinks to distribute data
or docker-compose to mount projects
Package: dataspace
DataspaceA simple api to explore, clean, transform and visualize dataFeaturesExplore data: describe, search and visualize raw dataClean and transform data: select, filter, normalize and reshape dataVisualize data: many kind of charts:books: Read thedocumentationDocData ioLoadFrom dfFrom csvFrom djangoExportExport csvInfosView dataShowCount dataCount nullCount zeroCount uniqueWuniqueSelectLimitUniqueCleanNullsDrop naDrop any nullsDrop all nullsFill nullsDatesTo dateTo tzdateFdateTimestampsConvert typesTo intTo floatTo strTo typeClean valuesStripStrip colsRoundvalsReplaceTransformDataframeSplitDropAddRenameKeepCopycolReverseValuesSortExcludeAppendMappendDiffmResample timeseriesResampleRsumRmeanCalculationsPercentDiffpDiffppDiffnDiffnpDiffmDiffmpCvarLregChartsOptionsAxisAltairBokehWHWhDraw chartsLinePointBarAreaSquareRuleTickBar numLine numPoint numHeatmapHistHlineInline apiWHWhMwPwColorOpacityTooltipToRxNoxNoyTitleColormapQcolormapSave imgGet htmlHtml headerReportingPrepare a reportReport pathStackExportSave pdfSave htmlThis api is:Minimalistic: short names, simple functionalites, minimal parametersPragmatic: focuses on raw efficiency rather than strictly idiomatic code while favouring static typingSimple stupid: keep it easy to understand for both code and apiInstallpip install dataspaceDependenciesDataframe:PolarsCharts:Altair, andHoloviewswithBokehExample notebooksExample notebooksare available├── chart
│   ├── area
│   ├── bar
│   ├── hline
│   ├── line
│   └── point
├── clean
│   ├── convert
│   ├── date
│   ├── drop_nulls
│   ├── fill_nulls
│   ├── strip
│   └── values
├── count
│   ├── count_empty
│   ├── count_null
│   ├── count_unique
│   └── count_zero
├── io
│   ├── export
│   └── load
├── select
│   ├── limit
│   ├── unique
│   └── wunique
└── transform
    ├── dataframe
    │   ├── add
    │   ├── copycol
    │   ├── drop
    │   ├── keep
    │   ├── rename
    │   └── split
    ├── diff
    │   ├── diffm
    │   ├── diffmp
    │   ├── diffn
    │   ├── diffnp
    │   ├── diffp
    │   └── diffpp
    ├── resample
    │   ├── rmean
    │   └── rsum
    └── values
        ├── append
        ├── exclude
        ├── reverse
        └── sortTestsTo run the tests:maketest
Package: dataspectra
Dataspectra^^^^^^^^^^^A python package to create interactive web visualizations.Requirements:=============- Python2.7Installation=============.. code-block:: bash$ pip install virtualenv$ virtualenv ~/virtualenv/ds$ source ~/virtualenv/ds/bin/activate$ pip install dataspectraDevelopment Notes=================Testing on your local computer without deploying++++++++++++++++++++++++++++++++++++++++++++++++- In order to test new functions you will need Google Cloud SDK- Test using dev_appserver.py --clear_datastore=yes app.yaml in the sample output.- Your website will now be visible on localhost:8080- Upload a sample subset of your data with- Once your code looks good. You can copy it to the source code in aefiles.- For running a quick test, use:.. code-block:: bash$ dataspectra developUpdating the documentation++++++++++++++++++++++++++- Written in Sphinx- Run the following in the docs directory... code-block:: bash$ make html- If using Microsoft Visual Studio, can preview by Command-Shift-R.Developing the package++++++++++++++++++++++.. code-block:: bash$ python setup.py developUploading package to pypi+++++++++++++++++++++++++.. code-block:: bash$ python setup.py sdist upload
Package: dataspike
Dataspike python client libraryThe official wrapper for Dataspike APIInstallationpip install dataspikeGetting startedLibrary offers well typed async API powered by pydantic and aiohttp.fromdataspikeimport*asyncwithApi("<YOUR_API_TOKEN>")asapi:verification=awaitapi.verification.create(checks=[CheckType.Passport,CheckType.Selfie])awaitapi.document.upload(verification.applicant_id,DocumentType.Passport,open('passport.jpg','rb'))awaitapi.document.upload(verification.applicant_id,DocumentType.Selfie,open('selfie.jpg','rb'))awaitapi.verification.proceed(verification.id)applicants=awaitapi.applicant.list()verifications=awaitapi.verification.list()ResourcesFull reference and documentation about available resources
can be found at ourofficial documentationCurrently library provides following resourcesAMLapi.amlApplicantapi.applicantVerificationapi.verificationSDKapi.sdkDocumentsapi.documentTimeoutsLibrary uses aiohttp ClientSession.
To pass timeouts use keyword arguments which will passed to ClientSession constructorCheck outaiohttp client referencefor details.ExamplefromdataspikeimportApiasyncwithApi('<API_TOKEN>',read_timeout=2)asapi:...Errorspydantic.ValidationErroris raised when type parameters not match with expected for API func.asyncio.TimeoutErroris raised if a timeout occurs.dataspike.errors.UnexpectedResponseStatusis raised whenever dataspike returns unexpected response status.Sync API wrapperWe recommend use async api directly. But if you really want to use sync api
library offersSyncApiwrapper. Take note it build on top of async API
and still use asyncio under the hood.ExamplefromdataspikeimportSyncApiwithSyncApi("<API_TOKEN>")asapi:applicants=api.applicant.list()