Saravanan666/pypi-org huggingface dataset:
Package: datafunctions
datafunctionsAutomatic (de)serialization of arguments and return values for Python functions.pip install datafunctions@datafunctionis a decorator which automatically deserializes incoming arguments of the decorated function and
serializes the return value. For example:fromdatetimeimportdatetimefromdatafunctionsimportdatafunction@datafunctiondefnext_year(dt:datetime)->datetime:returndt.replace(year=dt.year+1)assertnext_year("2019-01-02T00:00:00")=="2020-01-02T00:00:00"@datafunctionautomatically converts the string argument to a datetime object, and then
converts the returned datetime back to a string.This is useful for calling functions over a remote connection or from a different language - seeinstant_apiandinstant_clientfor example.More generally, the arguments and return value as seen from the outside the function
are basic JSON serializable objects - strings, dicts, etc.
They are converted to and from the correct types (as indicated by type annotations)
bymarshmallow. Common Python types as well as dataclasses (which may be nested)
are supported. For example:fromdataclassesimportdataclassfromdatafunctionsimportdatafunction@dataclassclassPoint:x:inty:int@datafunctiondeftranslate(p:Point,dx:int,dy:int)->Point:returnPoint(p.x+dx,p.y+dy)asserttranslate({"x":1,"y":2},3,4)=={"x":4,"y":6}To decorate a method, passis_method=True, e.g:classMyClass:@datafunction(is_method=True)defmethod(self,x:int)->int:...All parameters and the return value must have a type annotation,
except for the first argument whenis_method=True.
Variadic parameters (e.g.*argsor**kwargs) and positional-only parameters (before/)
are not allowed.If there is an exception deserializing or binding the arguments anArgumentErrorwill be raised with the underlying exception attached to__cause__.
Similarly aReturnErrormay be raised when trying to serialize the return value.For more manual control, use the methods:load_argumentsdump_argumentsload_resultdump_resultUnder the hood, the type annotations are gathered into adataclasswhich is then
converted into amarshmallowschema
usingmarshmallow_dataclass.
marshmallow handles the (de)serialization.Instances of this class have attributesparams_schemasandreturn_schemas,
each of which have the following attributes:dataclassschema_class: the marshmallow schema classschema_instance: a no-args instance of schema_class
Package: datafushion-plugins-python
DataFushion_Plugins_Python说明1.简介针对Python算法在DataFushion平台使用所给出的插件,主要用于规范化算法的输入输出2.使用Step1:引入datafushon包中的operation模块Step2:使用资源管理器进行数据拆解处理,并在其中实现自己需要实现的业务算法逻辑fromdatafushionimportoperationif__name__=='__main__':withoperation()asdestruction:input_data_struct_list=destruction.input_data_struct_listparam_map=destruction.param_mapparam_multiply=float(param_map['multiply'])data_result=[]forinput_data_structininput_data_struct_list:file_input_mapping=input_data_struct.file_input_mappingdata_list=input_data_struct.data_listfield_x=file_input_mapping['x'][0]field_y=file_input_mapping['y'][0]fordataindata_list:x=float(data[field_x])y=float(data[field_y])sum_result=param_multiply*(x+y)res={"sum":sum_result,"x":x,"y":y,}data_result.append(res)destruction.data_result=data_result注意:destruction为解构的HandleDataSet实体类，input_data_struct_list中包含了输入数据的封装,其类型为List其元素为HandleInputDataStruct类,包含的属性为file_type,file_path,file_input_mapping,data_list算法需要使用的是file_input_mapping和data_listdata_list是输入数据的字典列表file_input_mapping为输入数据字段的映射param_map为算法的参数字典在对数据进行业务算法处理完成后,需要将拆解的destruction中的data_result属性赋值为业务算法的最终数据结果3.模型DataFushion_Plugins_Python基础包中加入对PMML格式模型的依赖，以便后续需要在算法中读取已有的模型做准备frompypmmlimportModelif__name__=='__main__':model=Model.fromFile('model_file')foriteminmodel.inputFields:print(item.name)predict=model.predict({'sepalWidth':1.33,'sepalLength':2.44,'petalsWidth':5.87,'petalsLength':7.11,})print(predict)
Package: datafuzz
datafuzzA data-science library built for testing cleaning, schema validation and model robustness. Datafuzz messes up your data so you can test things before they go wrong in production.Free software: BSD licenseDocumentation:https://datafuzz.readthedocs.io.FeaturesTransform your data by adding noise to a subset of your rowsDuplicate data to test your duplication handlingGenerate synthetic data for use in your testing suiteInsert random “dumb” fuzzing strategies to see how your tools cope with bad dataSeamlessly handle normal input and output types including CSVs, JSON, SQL, numpy and pandasInstallationInstall datafuzz by running:$ pip install datafuzzRecommended use is with a proper Virtual Environment (learn more aboutvirtual environments <http://docs.python-guide.org/en/latest/dev/virtualenvs/>).For more details see Installation Instructions.ContributeIssue Tracker:https://github.com/kjam/datafuzz/issuesSource Code:https://github.com/kjam/datafuzzSupportIf you are having issues, please let reach out via the Repository issues.LicenseThe project is licensed under the BSD license.CreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.2 (2023-03-28)Update python versionsFix several nondeterministic bugs0.1.0 (2017-09-13)First release on PyPI.
Package: data-generation
data_generationAre you looking for a package that can generate high dimensional synthetic datasets?Do you need datasets with a grouped structure so that you can test your group lasso based formulation?Dont look more.data_generationis precisely what you need.Usage exampledata_equal = dgen.EqualGroupSize(n_obs=5000, ro=0.2, error_distribution='student_t', 
                                 e_df=5, random_state=1, group_size=10, non_zero_groups=3, 
                                 non_zero_coef=5, num_groups=7)

x, y, beta, group_index = data_equal.data_generation().values()data_different = dgen.UnequalGroupSize(n_obs=5000, ro=0.8, error_distribution='normal', e_loc=1, e_scale=4,
                                       random_state=2, tuple_group_size=(2, 4, 6, 8),
                                       tuple_number_of_groups=(5, 10, 15, 20),
                                       tuple_non_zero_coef=(1, 2, 3, 4),
                                       tuple_non_zero_groups=(1, 3, 5, 7))

x, y, beta, group_index = data_different.data_generation().values()For a deeper review and an explanation of the capabilities of this package we recommend to read the user_guide available in the repository.
Package: data-generator-postgresql-pkg
Put conf.cnf with such setting in one dir with data_generator and insert_to_db.
This is how to position settings in conf.cnfvarchar int date	//list of datatypes, spaces separated
100000			//count of lines in file
landing_zone/		//landing_zone path
done/			//done path
error/			//error path
log.txt			//log.txt file path and name
10			//how many file generate each hour
mydb			//database name
postgres		//username
qwop1290		//password
192.168.10.155		//host name
test users nana		//tables name, for what generate data
0.03			//how many files in total GB size add to postgres
insert_dummy_function	//name of postgresql function, what inserts data
Package: data-geo
Conversor GNSS a GeoJSONDataEste módulo permite convertir los datos generados por una estación GNSS Trimble y
emitido por GSOF a una estructura deGeojson.Se inicializa:fromdata_geoimportGeoJSONDatakwargs=dict(station="CODE"position=<<diccionarioECEF*ylat-lon-z>>)process=GeoJSONData(**kwargs**)Debe contener las tablas ECEF y VCV al menos para poder realizar la conversión.Se transforma set de datos.fordatainlista:print(process.manage_data(data))Registro logsSi ocurre algún error estos se registran en el log asignado alog_path
Package: dataget
DatagetDataget is an easy to use, framework-agnostic, dataset library that gives you quick access to a collection of Machine Learning datasets through a simple API.Main features:Minimal: Downloads entire datasets with just 1 line of code.Framework Agnostic: Loads data asnumpyarrays orpandasdataframes which can be easily used with the majority of Machine Learning frameworks.Transparent: By default stores the data in your current project so you can easily inspect it.Memory Efficient: When a dataset doesn't fit in memory it will return metadata instead so you can iteratively load it.Integrates with Kaggle: Supports loading datasets directly from Kaggle in a variety of formats.Checkout thedocumentationfor the list of available datasets.Getting StartedIn dataget you just have to do two things:Instantiate aDatasetfrom our collection.Call thegetmethod to download the data to disk and load it into memory.Both are usually done in one line:importdatagetX_train,y_train,X_test,y_test=dataget.image.mnist().get()This example downloads theMNISTdataset to./data/image_mnistand loads it asnumpyarrays.Kaggle SupportKagglepromotesthe use ofcsvfiles anddatagetloves it! With dataget you can quickly download any dataset from the platform and have immediate access to the data:importdatagetdf_train,df_test=dataget.kaggle(dataset="cristiangarcia/pointcloudmnist2d").get(files=["train.csv","test.csv"])To start using Kaggle datasets just make sure you have properly installed and configured theKaggle API. In the future we want to expand Kaggle support in the following ways:Be able to load any file thatnumpyorpandascan read.Have generic support for other types of datasets like images, audio, video, etc.e.gdataget.data.kaggle(..., type="image").get(...)InstallationpipinstalldatagetContributingAdding a new dataset is easy! Read our guide onCreating a Datasetif you are interested in contributing a dataset.LicenseMIT License
Package: datagif
A Python package for easily making clean animated gifs of data plots,datagifis built on top
ofseabornto manage the plot generation, andimageioto manage creating the animated gif.Example GIFCode for this example can be found in theexamples directory.How to usedatagifThe method you need is named after the package, so use the following import statementfromdatagifimportdatagifYour data will need to be in the form of apandasDataFrame, with at least three columns.
Select one to be the time dimension, and then the simplest example looks like the followingdatagif(plots='scatterplot',save_dir='/path/to/plot/dir',name='GIF',data=df,x='x',y='y',t='time')Customizing your GIFThere are four important arguments for customizing your GIF and the plots that make it up:plt_funcs(Functions that are normally called likeplt.title())seaborn_funcs(Functions that are normally called likesns.set_theme())seaborn_args(Arguments to add to the plot functions)imageio_args(Arguments for the imageioget_writer()method)For the_funcsarguments, these should be in the form of a dict where the keys are strings of the
names of the functions you want to call, and the values should be the arguments you want to pass to
the functions. These values can themselves bea single value, like a stringa list to be unpacked and passed to the functiona dictionary to be unpacked and passed to the function as keyword argumentsFor the_argsarguments, these should be in the form of a dict that will be passed to the
plotting and imageio functions as keyword arguments.Using multiple plots in one GIFAs in the example above, you can layer multiple plots in one GIF! To do this, simply make theplotsargument a list of valid strings. Then also make thex,y,seaborn_argsarguments
into lists of valid values the same length as theplotslist, and the nthentry of
these other arguments will be used for the nthplot.Helpful TipsIt is strongly recommended to use thetight_layoutmethod in theplt_funcsargument.
This will help prevent small differences between the plots produced, leading to a much
cleaner GIF.
Package: datagov-harvesting-logic
datagov-harvesting-logicThis is a library that will be utilized for metadata extraction, validation,
transformation, and loading into the data.gov catalog.FeaturesExtractGeneral purpose fetching and downloading of web resources.Catered extraction to the following data formats:DCAT-USValidationDCAT-USjsonschemavalidation using draft 2020-12.LoadDCAT-USConversion of dcat-us catalog into ckan dataset schemaCreate, delete, update, and patch of ckan package/datasetRequirementsThis project is usingpoetryto manage this project. Installhere.Once installed,poetry installinstalls dependencies into a local virtual environment.TestingCKAN load testingCKAN load testing doesn't require the services provided in thedocker-compose.yml.catalog-devis used for ckan load testing.Create an api-key by signing into catalog-dev.Create acredentials.pyfile at the root of the project containing the variableckan_catalog_dev_api_keyassigned to the api-key.Run tests with the commandpoetry run pytest ./tests/load/ckanHarvester testingThese tests are found inextract, andvalidate. Some of them rely on services in thedocker-compose.yml. Run using dockerdocker compose up -dand with the commandpoetry run pytest --ignore=./tests/load/ckan.If you followed the instructions forCKAN load testingandHarvester testingyou can simply runpoetry run pytestto run all tests.Comparison./tests/harvest_sources/ckan_datasets_resp.jsonRepresents what ckan would respond with after querying for the harvest source name./tests/harvest_sources/dcatus_compare.jsonRepresents a changed harvest sourceCreated:datasets[0]+ "identifier" = "cftc-dc10"Deleted:datasets[0]- "identifier" = "cftc-dc1"Updated:datasets[1]- "modified": "R/P1M"+ "modified": "R/P1M Update"datasets[2]- "keyword": ["cotton on call", "cotton on-call"]+ "keyword": ["cotton on call", "cotton on-call", "update keyword"]datasets[3]"publisher": {"name": "U.S. Commodity Futures Trading Commission","subOrganizationOf": {-   "name": "U.S. Government"+   "name": "Changed Value"}
}./test/harvest_sources/dcatus.jsonRepresents an original harvest source prior to change occuring.
Package: datagovsg
This is an unofficial Python package for interacting with APIs available atData.gov.sg.Installing the packageInstall the package usingpip:pip install datagovsgUsing the packageThe main steps are:Import a class.Instantiate an object from the class.Call a function on that object.For more information,refer to the documentation.Usage overviewInteracting withData.gov.sg’s API is done through one of four clients, where each client corresponds with a “set” of endpoints. (Data.gov.sgdoesn’t categorise its endpoints by set, but it can be assumed from the endpoints’ path directories.)The four clients are:Ckan,Environment,TechnologyandTransport.Each client contains several public functions, one function per endpoint. A function’s name is the same as its corresponding endpoint’s ending path.Most functions accept named arguments, where an argument corresponds with a parameter that the endpoint accepts.Why have separate clients instead of one single client?Without knowing howData.gov.sg’s API will evolve, and noticing that the endpoints were themselves already partitioned into “sets”, it seemed like a good idea to keep each set of endpoints in its own contextual client. This allows for each “set” of endpoints to be customised on their own, e.g. theEnvironmentendpoints allow for either a date or date-time to be specified, whereas theTransportendpoints don’t.ReferenceData.gov.sg’s Developer Guide
Package: datagram
This is a small package with some functions that help me to handle data
in a fast and easier way. The main goal is to fit data without making
a mess.Although of course there’s nothing new here, the classes Histogram and
Distribution here present seem to me nicer to work with in comparison
with other frameworks. These two classes are the core of the package.Class Histogram allows fast generation of pdfs and cdfs, while data can
be added to the object at any time. The package is thought for discrete
data, usually sparse, so it is stored in a dictionary.Class Distribution is a wrapper of distribution functions (pdf, cdf,
likelihood, etc.) which is a counterpart of Histogram, used in the fitting
process.Copyright 2013 Jose Maria Miotto
Package: datagrepper
datagrepperDatagrepper is a web application and JSON API to retrieve historical messages sent via Fedora Messaging.Datanommeris a seperate project and service that consumes messages from the Fedora Messaging queue and puts them in a database. These messages is what datagrepper queries.Datagrepper is curently running in production athttps://apps.fedoraproject.org/datagrepper/Development EnvironmentVagrant allows contributors to get quickly up and running with a datagrepper development environment by automatically configuring a virtual machine.The datagrepper Vagrant environment configures configures and enables a datanommer service and database. The datanommer instance is configured to be empty when first provisioned, but to consume messages from the stage Fedora Messaging queue.Install vagrantTo get started, run the following commands to install the Vagrant and Virtualization packages needed, and start the libvirt service:$ sudo dnf install ansible libvirt vagrant-libvirt vagrant-sshfs vagrant-hostmanager
$ sudo systemctl enable libvirtd
$ sudo systemctl start libvirtdCheckout and ProvisionNext, check out the datagrepper code and run vagrant up:$ git clone https://github.com/fedora-infra/datagrepper
$ cd datanommer
$ vagrant upInteracting with your development datagrepperAfter successful provisioning of the Datagrepper vagrant setup, the datagrepper web application will be accessible from your host machine's web browser athttp://datagrepper.test:5000/Using the development environmentSSH into your newly provisioned development environment:$ vagrant sshThe vagrant setup also defines 4 handy commands to interact with the service that runs the datagrepper flask application:$ datagrepper-start
$ datagrepper-stop
$ datagrepper-restart
$ dataprepper-logsAdditionally, the following commands are also available for interacting with the datanommer service:$ datanommer-consumer-start
$ datanommer-consumer-stop
$ datanommer-consumer-restart
$ datanommer-consumer-logsRunning the testsDatanommer is comprised of 3 seperate modules in this single repository. There is a handy script in the top directory of this repo to run the tests on all 3 modules:$ ./runtests.shHowever, tests can also be run on a single module by invotking tox in that modules' directory. For example:$ cd datanommer.models/
$ toxNote, that the tests use virtual environments that are not created from scratch with every subsequent run of the tests. Therefore,when changes happen to dependencies, the tests may fail to run correctly. To recreate the virtual envrionments,  run the tests commands with the-rflag, for example:$ ./runtests.sh -ror$ cd datanommer.models/
$ tox -r
Package: datagrid_gtk3
SYNOPSISMVC framework for working with the Gtk3 TreeView widget.
Includes classes for file browsing and displaying SQLite databases.MOTIVATIONCreated on the account of the reoccurring need to display and interact with
contents of SQLite databases in Gtk applications.History0.1.x (2015-08-11 - 2015-08-26)Bug fix releases0.1.2 (2015-04-28)Initial public release.
Package: data-grid-surface
DATA-GRID-SURFACESDK to communicate with data-grid API service.
It uses the API service and it's end-points to determine if the given emails or passwords have been compromised.InstallationInstall data-grid-surface SDK:pip install data-grid-surfaceUsing data-grid-access sdkImport DataGrid class from libraryfrom data_grid_surface.data_grid import DataGridYou will need to provide username and password parameters to DataGrid class constructor. These are credentials for data-grid API service.NOTE: Passwords and emails are hashed with SHA256 algorithm before being sent to the API service.DataGrid methodsDataGrid methods return dictionary as a result.You can pass raw email/password or its hashed value. If you are passing hashed value you need to hash it with SHA256 algorithm and encode it in base64 format.Methods:check_email(email, is_hashed)email<String>is_hashed<Boolean>default value is Truecheck_password(password, is_hashed)password<String>is_hashed<Boolean>default value is Truecheck_phone_number(phone_number, is_hashed)phone_number<String>is_hashed<Boolean>default value is TrueUse example:from data_grid_surface.data_grid import DataGrid

dg = DataGrid(
    username=<datagridapiusername>, 
    password=<datagridapipassword>
)
res = dg.check_email('email@example.com', False)res = dg.check_password('passwordexample', False)res = dg.check_phone_number('+38164....', False)Response:{
    'status': 'success', 
    'data': {
        'exposed': True|False,
        'breach_source': 'facebook'
    }
}
Package: datagrowth
DATAGROWTHData Growth is a Django application that helps to gather data in an organized way. With it you can declare pipelines
for the data gathering and preprocessing as well as pipelines for filtering and redistribution.InstallationYou can install Datagrowth with your Django application by runningpipinstalldatagrowthGetting startedCurrently there are two major use cases.
TheResourcesprovide a way to uniformly gather data from very different sources.Configurationsare a way to store and transfer key-value pairs,
which your code can use to act on different contexts.Follow these guides to get an idea how you can use Datagrowth:ResourcesConfigurationsRunning the testsThere is a Django mock project inside the tests directory of the repository.
You can run these tests by running this inside that directory:pythonmanage.pytestAlternatively you can execute the tests against multiple Django versions by running:tox
Package: datagym
Python API Wrapper for DataGym (https://www.datagym.ai) - the AI-assisted image annotation tool.
You can easily access your Projects, Datasets, Images and labeled data
from the DATAGYM backend through this package.DATAGYM enables data scientists and machine learning experts to label images up to 10x faster.
AI-assisted annotation tools reduce manual labeling effort, give you more time to finetune ML models and speed up your go to market of new products.Free software: BSD licenseDocumentation:https://docs.datagym.ai/documentation/python-api/GitHub:https://github.com/datagym-ai/datagym-pythonRequirementsPython Version:Python 3.6, 3.7, 3.8, or laterDataGym Account:If you haven’t got a DataGym account yet, pleaseSignUp.Initial Project:Setup your first Projectand/or label some images in your Dummy ProjectAPI Token:In order to access your Dummy Project via our Python API you need tocreate an API TokenFirst StepsInstall the package via pip:user@machine:~$pipinstalldatagymCreate your first DATAGYM API Client:fromdatagymimportClientclient=Client(api_key="<YOUR_API_KEY>")FeaturesFetch Project, Datasets, ImagesExport labeled data from ProjectsImport labeled data into ProjectsCreate DatasetsAdd Datasets to ProjectsUpload/Download imagesUpload of coco labelsUpload of label configuration
Package: datahand
Data HandDataHand is a collection of algorithms to read and handle data for research (e.g., Web of Science, USPTO) in Python.DataHand allows to read a variety of typical sources used for research and convert them into a pandas DataFrame.InstallationTo install DataHand, we recommend to use PyPI:pip install datahandFirst steps1. Read data from Web of Sciencefrom datahand import read_wos

df = read_wos("file/to/path.txt")
Package: datahandler-bhanu
datahandler(python)Handling Missing values in a given data set and imputing them in python.Install this library using pip in cmd/powershell (ADMIN) ----> pip install datahandler-bhanuComes with command line support. Use directly from cmd/powershell by typing python filename.py datafilename.csv and wallah!
Github Link ->https://github.com/eternaldemon/datahandlerPyPi Link ->https://pypi.org/project/datahandler-bhanu/Libraries Used --> Pandas,sklearn,sys,numpyYou can import it directly using import datahandler and use the function datahandler as datahandler.datahandler The updated data will be exported to a new file called data_new.csv