Saravanan666/pypi-org huggingface dataset:
Package: datadreamer.dev
https://datadreamer.devPrompt. Generate Synthetic Data. Train & Align Models.DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. It is designed to be simple, extremely efficient, and research-grade.Installationpip3 install datadreamer.devdemo.pyResult ofdemo.pySee thefull demo scriptSee thesynthetic datasetandthe trained model🚀 For more demonstrations and recipes see theQuick Tourpage.With DataDreamer you can:💬Create Prompting Workflows: Create and run multi-step, complex, prompting workflows easily with major open source or API-based LLMs.📊Generate Synthetic Datasets: Generate synthetic datasets for novel tasks or augment existing datasets with LLMs.⚙️Train Models: Align models. Fine-tune models. Instruction-tune models. Distill models. Train on existing data or synthetic data.... learn more about what's possible in theOverview GuideDataDreamer is:🧩Simple: Simple and approachable to use with sensible defaults, yet powerful with support for bleeding edge techniques.🔬Research-Grade: Built for researchers, by researchers, but accessible to all. A focus on correctness, best practices, and reproducibility.🏎️Efficient: Aggressive caching and resumability built-in. Support for techniques like quantization, parameter-efficient training (LoRA), and more.🔄Reproducible: Workflows built with DataDreamer are easily shareable, reproducible, and extendable.🤝Makes Sharing Easy: Publishing datasets and models is simple. Automatically generate data cards and model cards with metadata. Generate a list of any citations required.... learn more about themotivation and design principles behind DataDreamer.CitationPlease cite theDataDreamer paper:@misc{patel2024datadreamer,title={DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows},author={Ajay Patel and Colin Raffel and Chris Callison-Burch},year={2024},eprint={2402.10379},archivePrefix={arXiv},primaryClass={cs.CL}}ContactPlease reach out to us viaemail (ajayp@upenn.edu)or onDiscordif you have any questions, comments, or feedback.Copyright © 2024,Ajay Patel. Released under theMIT License.Thank you to the maintainers atHugging FaceandLiteLLMfor accepting contributions necessary for DataDreamer and providing upstream support.Funding AcknowledgementsODNI, IARPA:This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Package: data-drift-detector
Data Drift DetectorThis package contains some developmental tools to detect and compare statistical differences between 2 structurally similar pandas dataframes. The intended purpose is to detect data drift - where the statistical properties of an input variable change over time.We provide a classDataDriftDetectorwhich takes in 2 pandas dataframes and provides a few useful methods to compare and analyze the differences between the 2 datasets.InstallationInstall the package with pippipinstalldata-drift-detectorExample UsageTo compare 2 datasets:fromdata_drift_detectorimportDataDriftDetector# initialize detectordetector=DataDriftDetector(df_prior=df_1,df_post=df_2)# methods to compare and analyze differencesdetector.calculate_drift()detector.plot_numeric_to_numeric()detector.plot_categorical_to_numeric()detector.plot_categorical()detector.compare_ml_efficacy(target_column="some_target_column")You may also view an example notebook in the following directoryexamples/example_usage.ipynbto explore how it may be used.
Package: data-drift-detector-mightyhive
About The PackageThe package is a wrapper of tensorflow data validation for our specific needs.
It can analyze training data and serving data to compute desscriptive
statistics, infer a schema, and detect anomalies.Dependenciestensorflow-data-validationInstallationpipinstalldata-drift-detectorUsageInitialize a Harvest client:# The Dataset, TrainDataset, ServeDataset can be initialized with different methods.train=TrainDataset.from_GCS()train=TrainDataset.from_bigquery()train=TrainDataset.from_dataframe()train=TrainDataset.from_stats_file()Populate the class variables and submit.# Get training dataset schemaschema=train.schema_dict()
Package: datadriven
datadrivendatadriven allows you to run a test case multiple times with different data. It was designed to work
withunittest.TestCasetest methods, any test runner (nose, nose2, unittest, unittest2 and
pytest), in Python 2.7 or 3.7+.UsageimportdatadrivenclassSomeTestCase(unittest.TestCase):@datadriven.datadriven(caseA=datadriven.Args(first="foo"),caseB=datadriven.Args("hello",second="world"),)deftest_method(self,first,second=None):...This is the equivalent of writing:importdatadrivenclassSomeTestCase(unittest.TestCase):deftest_method(self,first,second=None):...deftest_method_caseA(self):self.test_method("foo")deftest_method_caseB(self):self.test_method("hello","world")The only difference is that when wrapped with@datadriven, the testtest_methodwon't be really executed.LicenseApache-2.0 License
Package: datadriver
# DataDriver
DataDriver is a unified set of tools for accelerating the industrialization of Data Science projects.## Our vision
We noticed that the most impactful parameter on the success of a data science project was the ability to go to the production phase. We focused on making a tool that facilitate the transition from exploration to production rather than a tool to help the data scientist to explore data or make good models.We think that the sooner we can put a model in a production environment, the better our project chances are at being succesful. The true difficulty of a data science project lies not in solving the problem using a machine learning model but in facing the complexity of the production setting.## What it is
It packages together [ddapi](https://github.com/octo-technology/ddapi) and [ddui](https://github.com/octo-technology/ddui).We made an API (pyddapi) that lets data scientists use the technologies they use in exploration be compatible with the industrialization phase. We rely on Airflow to schedule, execute reliably and monitor jobs.
We also developped an Airflow plugin for Data Science (ddui) that, combined with ddapi, allows you to visualize the output of any task with metrics and charts.## Installation`bash pip install datadriver `## Usage
Go to the root of your project directory and launch DataDriver with a single command.`bash datadriver `
Package: dataduct
Dataduct is a wrapper built on top of AWS Datapipeline which makes it easy to
create ETL jobs. All jobs can be specified as a series of steps in a YAML file
and would automatically be translated into datapipeline with appropriate
pipeline objects.Documentation and DetailsDocumentation and more details can be found athttp://dataduct.readthedocs.org/en/latest/LicenseCopyright [2014] [Coursera]Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: dataduct-flywire
Dataduct is a wrapper built on top of AWS Datapipeline which makes it easy to
create ETL jobs. All jobs can be specified as a series of steps in a YAML file
and would automatically be translated into datapipeline with appropriate
pipeline objects.Documentation and DetailsDocumentation and more details can be found athttp://dataduct.readthedocs.org/en/latest/LicenseCopyright [2014] [Coursera]Licensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: data-dummy
Data dummy generatorRandom name, username, email, phone, address, ipv4 generatorInstallationinstall with pip::pip install data-dummy==1.0UsageHere are examples of all current features:>>> import dummy
>>> dummy.name()
'Yuliana Sudradjat'
>>> dummy.username()
'Yuliana'
>>> dummy.email()
'Bernard@gmail.com'
>>> dummy.address()
'Jl. Dr A Rivai Bukittinggi Telp: 0752-21720'
>>> dummy.phone()
'+62812-0989-0128'
>>> dummy.ipv4()
'123.124.5.12'LicenseThis project is released under anMIT License.Src datadata.name (the following files are public domain in indonesia )data.address (address of covid19 care referral hospital in indonesia)available on PyPI:https://pypi.org/project/data-dummy/
Package: data-duper
data-duperdata-duper is a tool to replicate the structure of private or protected data for testing.What does it solve?When testing the data handling of software, it is best to use data as similar to the real data as possible - without revealing sensitive information to the test environment. This is where data-duper comes into play. It allows you to create an authentic replicate of your private or protected data.How to get it?The source code is currently hosted on GitHub at:https://github.com/kjanker/data-duper.Binary installers for the latest released version are available at thePython
Package Index (PyPI).What does it do?data-duper works like a learning model. You train the duper on your real data and, afterwards, generate a new data set of arbitrary size. The new data set - or dupe - has the same structure as the real data, i.e., columns, dtypes, as well as string composition and distribution of numerical values. Occurrences of NA values are ignored by default but can optionally be included as well.Methodsnumerical values (float, int, datetime) are drawn from an interpolated empirical distributionidentifier strings of fixed length and structure are replicated with regular expressionsfeatures with only few values (category, bool) are redrawn according to their occurrenceLimitationsvalue distributions are replicated as draw probability. Thus, for small dupe sets the realized distribution may differ slightlycorrelations between columns are not replicated (this ensures real data is better obscured)descriptive strings like notes, names, etc are not obscured but reshuffledHow can I use it?You simply initialize a newDuperinstance, fit it on your real datadf_real, and make a data dupedf_dupeof desired sizen.fromduperimportDuperduper=Duper()duper.fit(df=df_real)df_dupe=duper.make(size=10000)Open issuesinclude optional correlations between selected rowsimprove algorithm of regex duperGet in touchDon't hesitate to contact me if you like the idea and want to get in touch.
Package: datadynamics
DatadynamicsDatadynamics is a Python library and environment for simulating (multi-agent) data collection dynamics. The library is built on top ofPettingZooand is distributed under theBSD 3-Clause License.The documentation is available atlfwa.github.io/datadynamics/.InstallationDatadynamics can be installed via PyPI from Python 3.10 and higher:$pipinstalldatadynamicsAlternatively, you can install from source by downloading thelatest releaseor by cloning theGitHub repository, navigating into the directory, and installing viaPoetry:poetry install.UsageVisit the documentation site atlfwa.github.io/datadynamics/for full usage guides.Quick Start# See tutorials/collector/wrapper_example.pyfromdatadynamics.environmentsimportcollector_v0fromdatadynamics.policiesimportgreedy_policy_v0env=collector_v0.env(n_points=300,n_agents=2,max_collect=[120,180],render_mode="human")policy=greedy_policy_v0.policy(env=env)env.reset()foragentinenv.agent_iter():observation,reward,termination,truncation,info=env.last()action=policy.action(observation,agent)env.step(action)
Package: data-email-client
data_email_clientdata_email_clientis a Python package for downloading data files from emails using the imap protocol (SSL-compatible)installationpip install data_email_clientexamplessee script docstrings for more usage info.spidar data files:>>>fromdata_email_clientimportmailer>>>fromgetpassimportgetpass>>>server='outlook.office365.com'# 'imap.gmail.com' for gmail>>>username='data-email@my-domain.com'>>>password=getpass()...>>>imap=mailer(server=server,username=username,password=password)>>>data_boxes=[mforminimap.mailboxesif'data'inm]>>>imap.search_for_messages(text='spidardatanotification@nrgsystems.com',area='from',folder=data_boxes)>>>imap.download_attachments(out_dir='/path/to/data/',extension='csv',delete=False)symphonie data emails...>>>body_text='SymphoniePRO Logger data attached.'# 'Wind Data attached.' for older logger types>>>imap=mailer(server=server,username=username,password=password)>>>data_boxes=[mforminimap.mailboxesif'data'inm]>>>imap.search_for_messages(text=body_text,area='body',folder=data_boxes)>>>imap.download_attachments(out_dir='/path/to/data/',extension='rld',delete=False,archive_folder='INBOX/Archive')
Package: dataembedder
Data EmbedderEmbeds data and meshes in a host scaffold using Zinc.LicenseCopyright 2022 University of Auckland

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: dataengine
# Data EngineData Engine is a general purpose python package for data engineering.[![codecov](https://codecov.io/gh/leealessandrini/dataengine/branch/main/graph/badge.svg)](https://codecov.io/gh/leealessandrini/dataengine)## Development### Setting up a python virtual environmentCreate the virtual environment with any name you’d like (in this example we’ll use “myenv”`bash python-mvenv myenv `Activate the environemt`bash source myenv/bin/activate `Install the package requirements`bash pip install-rrequirements.txt `When you’re done using the environment you can deactivate it with the following command:`bash deactivate `
Package: data-entry2
data_entry2.pyA spreadsheet interface for data entry in Python/JupyterRequires: ipydatagridTo use:import data_entry2

  de = data_entry2.sheet("name")This creates or loads name.csv and presents as a "live"
spreadsheet-like interface. This is for data entry only, no
calculations are possible in the spreadsheet.Put a # in front of a row to comment it out
Put a $ in front of a column name to comment out the column (useful for
non-numeric columns)Simple expressions can be evaluated, eg: =1+3/np.sqrt(3)Generate Vectors generates a series of arrays for each non-commented
column.Two pandas data frames are also created:one is self.df that contains just the data in the vectors.the other, self.fdf is strings that has everything in the whole table.To take a snapshot of a current sheet and generate a new one:
de2 = data_entry2.sheet_copy("old", "new")if new.csv exists, it gets loaded, if not, old.csv is copied to new.csv.Add Row and Add Column do what they claim. Toremoveunwanted
empty rows and columns, just re-run the cell that sets up the
sheet. empty rows and columns will be deleted.Copy & Paste only work for single cells.To edit a cell - click and start typing. If you want to edit a current
value: try double clicking - doesn't always take, might need a few tries.
Or click and press F2.After editing a cell, enter moves to next cell down, probably want Tab to
move to next cell to right. (shift tab goes left).There is a top row of cell headers shown that allows: column
widths to be adjusted, and filters and sorting. Probably the
sorting and filters should not be available. Display of these
headers can be disabled by setting: header_visibility='row' in the
call to DataGrid, but then the widths can't be adjusted...
Can always restore the correct order by sorting key by ascending.
To make that work there are some extra spaces included in front of
the Variable and Units row headersIn order to help avoid disasterous data loss: every time Generate
Vectors is pressed, a snapshot of the current sheet is saved in
csv_backups (as long as it is different from the most recent
backup).  You can poke around in there and make copies of the
backups, rename them and move up to where the notebook can use it.BUGS: the sheet will only render when the notebook is exported if
Settings->Auto Save Widget State is toggled on.To output all sheets: call
display_sheets()
As long as each sheet was created with a different variable name,
they will all be displayed.
Package: data-entry-demo
Package to simulate manual data entry or test against other dataHow to install the package locally usingpipThe package is written to be installed just like any other pip package locally.  You may need to install some of the dependent packages necessary, since it doesn't currently have the ability to look for and install dependent packages.  One simply downloads the.zipfile and saves it (ideally where your other python packages are saved).  Then unzip, and use your command line tool to navigate into thedata_entry_demo/folder.  Install by typingpip install .and wait for the package to be built.  You can confirm by typingpip freezein your command prompt and verify thatdata-entry-demo-0.0.1is in the list of packages installed.  You can also open python and confirm thatfrom data_entry_demo import CHRDataEntryDemoruns without an error.
Package: data-enum
Data EnumsAn alternative to the built-in Pythonenumimplementation. Tested in Python 3.6 and above.Data enums allow you to:Associate data with enum members without using tuple-based initializationUse an intuitive initialization syntax to define membersDefine pure enums without usingauto()Define value-based enums without storing them as class attributesDefine secondary unique keys & use them to look up enum membersUse classmethod syntax (.get(…)) to look up members, instead of using initializersUsageInstall via PyPI:pip install data-enumMinimal usage:from data_enum import DataEnum

class Currency(DataEnum):
    data_attributes = ('symbol', 'name')

Currency('CAD', symbol='$', name='Canadian dollar')
Currency('USD', symbol='$', name='United States dollar')
Currency('EUR', symbol='€', name='Euro')Access the members by value:Currency.get('USD')
Currency.get('CAD')
Currency.get('EUR')Store the members as attributes:class Currency(DataEnum):
    data_attributes = ('symbol', 'name')

Currency.CAD = Currency('CAD', symbol='$', name='Canadian dollar')
Currency.USD = Currency('USD', symbol='$', name='United States dollar')
Currency.EUR = Currency('EUR', symbol='€', name='Euro')Use a custom attribute as the primary ID:class Currency(DataEnum):
    primary_attribute = 'code'
    data_attributes = ('symbol', 'name')

Currency('CAD', symbol='$', name='Canadian dollar')
Currency('USD', symbol='$', name='United States dollar')
Currency('EUR', symbol='€', name='Euro')Use integers as primary IDs:class Door(DataEnum):
    data_attributes = ('description',)

Door(1, description='Door #1')
Door(2, description='Door #2')
Door(3, description='Door #3')

d2 = Door(2)  # returns Door(2, description='Door #2')

int(d2)  # returns 2Or, skip primary IDs altogether for a pure enumeration:from data_enum import DataEnum

class Currency(DataEnum):
    data_attributes = ('symbol', 'name')

Currency.CAD = Currency(symbol='$', name='Canadian dollar')
Currency.USD = Currency(symbol='$', name='United States dollar')
Currency.EUR = Currency(symbol='€', name='Euro')Access the attached data:print(Currency.USD.name)  # prints 'United States dollar'
print(Currency.EUR.symbol)  # prints '€'

print(Currency.USD)  # prints 'USD'Compare directly:Currency.USD == Currency.CAD  # returns False
Currency.EUR == Currency.EUR  # returns TrueEnforce unique secondary attributes:class Currency(DataEnum):
    # Use a tuple with the second value as True for unique keys
    data_attributes = (('symbol', True), 'name')

# Throws ValueError
Currency('CAD', symbol='$', name='Canadian dollar')
Currency('USD', symbol='$', name='United States dollar')Look up members by unique secondary attributes:Currency.get(symbol='€')  # returns Currency.EUR
Currency.get(symbol='&')  # throws MemberDoesNotExistErrorLook up with members with defaults:Currency.get('ZZZ', Currency.USD)  # returns Currency.USD
Currency.get('ZZZ', default=Currency.USD)  # returns Currency.USD
Currency.get(symbol='&', default=Currency.USD)  # returns Currency.USDTesting, etc.Install requrements:make installSort imports (Requires Python >= 3.6):make formatLint (Requires Python >= 3.6):make lintTest:make test
Package: dataer-tool
dataer_toolthe tool for dataerDevelopLibray使用命令nbdev_build_lib可以生成python库Document如果修改过nbs下面的notebook, 使用命令nbdev_build_docs可以更新文档目录映射关系在docs/sidebar.json中可以设置, 首页index.html对应的notebook是nbs/index.ipynbAboutThis project generate fromnbdevAs templatesed -i 's/dataer_tool/YOUR_LIB_NAME/g' `grep -inr dataer_tool -rl .`
mv dataer_tool YOUR_LIB_NAME
Package: dataesg
DATAESGDataesg PythonThis is the official documentation for Dataesg’s Python Package. This package is compatible with python v3+.InstallationTo complete the installation of Dataesg’s Python Package, execute either of the following pip commands from terminal:$pipinstalldataesg
$pip3installdataesgAPI ConfigurationsPlease use the provided API keys to be able to access our services.
Example API Key: 32e6f6c4-33b1-4156-8ae6-8t92e7ab66ccUsing Python to retrieve ESG data$importdataesg
$dataesg.connect(‘32e6f6c4-33b1-4156-8ae6-8t92e7ab66cc’)$dataesg.get_sample(metric=[‘Scope_1’,‘Scope_2’],return_type=‘pandas’)$dataesg.get_by_ticker(ticker=‘MMM’,return_type=‘pandas’)$dataesg.get_by_isin(isin=[‘US88579Y1010’,’SE0000103814’],return_type=‘pandas’)$dataesg.get_by_cusip(cusip=‘88579Y101’,return_type=‘numpy’)$dataesg.get_by_sedol(sedol=‘2595708’,return_type=‘numpy’)Parameters$metrics-stringorlistofstrings(default:returnsallmetrics)$return_type-either‘pandas’or’numpy’(default:‘pandas’)$api_key-IfnotconnectedpreviouslyLicenseMIT License
Package: dataexpectations
data_expectationsIs your data meeting your expectations?Data Expectations is in Alpha - interface and usage patterns are being stablizedA delarative approach to asserting qualities of your datasets. Instead of tests likeis_sortedto determine if a column is ordered, the expectation iscolumn_values_are_increasing. Most of the time you don't need to knowhowit got
like that, you are only interestedwhatthe data looks like.Expectations can be used alongside, or in place of a schema validator, however they
are not intended to perform validation of the data in a dataset, not just the structure
of a table.Expectationsexpect_column_to_existexpect_column_names_to_match_setexpect_column_values_to_not_be_nullexpect_column_values_to_be_of_typeexpect_column_values_to_be_in_type_listexpect_column_values_to_be_betweenexpect_column_values_to_be_increasingexpect_column_values_to_be_decreasingexpect_column_values_to_be_in_setexpect_column_values_to_match_regexexpect_column_values_to_match_likeexpect_column_values_length_to_be_beexpect_column_values_length_to_be_betweenInstallpip install data_expectationsif data doesn't match, I'm not angry, I'm just disappointed.