Saravanan666/pypi-org huggingface dataset:
Package: data-pypeline
PypelineThis is a package for creating iterative data processing pipelines. Note that
this is NOT a general purpose stream processing library. It is only designed as
being a low overhead and simple-to-setup stream processing library. So for
large scale production applications, use something like kafka instead.WarningThis library is still at an ALPHA stage. So things may not work as intended
and the api is not final!Trivial Examplefrompypelineimportbuild_action,Pypeline,ForkingPypelineExecutor,wrapimportasyncioasyncdefstep1():results=[]foriinrange(1000):results.append(wrap(i))returnresultsasyncdefstep2(i):returni*10asyncdefstep3(i):returni+1asyncdefrun_pipeline():pypeline=Pypeline()# Adding actions to the pipelinepypeline.add_action(build_action("Step1",step1))\.add_action(build_action("Step2",step2))\.add_action(build_action("Step3",step3,serialize_dir="./example"))# Serialize results so future runs will skip this step entirelyresults=awaitpypeline.run(executor=ForkingPypelineExecutor())# Custom executor that avoids the GIL# Results are wrapped in a utility namedtuple, so let's flatten it.results=[r.args[0]forrinresults]returnresultsresults=asyncio.get_event_loop().run_until_complete(run_pipeline())forresultinresults:print(result)
Package: dataqa
DataQADataQA is a tool to perform AI model quality assessment (QA) using an interactive app that can be shared with technical and non-technical members of your team.TODO: Add a gif.The official documentation page is at:docs.dataqa.ai.Installationpip install dataqaQuick startStep 1: create an accountGo tohttps://app.dataqa.ai/and follow the steps to create your first project. Once your account and your first project have been created, you will see a screen such as this one:TODO: Add screenshot of the screen with the publish stringYou will need this key later in order to be able to create your first QA app. You can always come back to this page to find it.Step 2: Publish your dataCreating your first shareable QA app is as simple as this:importpandasaspdfromdataqa.publishimportDataQAdataqa=DataQA()dataqa.login()# Prompt username and passworddf=pd.DataFrame([[1,"Laptop",1600],[2,"Mouse",10]],columns=["id","product","price"])dataqa.publish(PROJECT_ID,df)ThePROJECT_IDis the hash string on the dataqa project page.Step 3: Use the UI to explore your dataTODO: add screenshot or GIF
Package: dataql
Python backend for “Data Query Languages” (like GraphQL and othersWhat is it?dataqlis for “Data Query Language”. It allows to query data in a
simple way.It is heavily inspired byGraphQL, from
FacebookI didn’t want to force people to think “graph”, and I chose a language
that is different in some ways. But this library is written with a base,
and we provide a generic parser, but other parsers could easily be
written!How it works?For example, the default generic parser included,DataQLParser,
allows to ask for data with this example query:User.get('Elon Musk') {
    name,
    birthday.strftime('%x'),
    companies[
        name,
        date:created_year,
    ]
}And to get data like that:{'name':'Elon Musk','birthday':'06/28/71','companies':[{'name':'Paypal','date':1999},{'name':'Space X','date':2002}]}The main use is for an API, letting the client asking what it really
needs, in only one http query, without having to update the API
endpoints.The only thing to do on the server side is to define, via a registry,
what objects and attributes are allowed.
Package: dataquality
dataqualityThe Official Python Client forGalileo.Galileo is a tool for understanding and improving the quality of your NLP and CV data.Galileo gives you access to all of the information you need, at a UI and API level, to continuously build better and more robust datasets and models.dataqualityis your entrypoint to Galileo. It helps you start and complete the loop of data quality improvements.ToCGetting StartedCustom IntegrationsNo labels? No problemProgrammatic AccessContributingGetting StartedInstall the package.pipinstalldataqualityCreate an account atGalileoGrab yourtokenGet your dataset and analyze it withdq.auto(You will be prompted for your token here)importdataqualityasdqdq.auto(train_data="/path/to/train.csv",val_data="/path/to/val.csv",test_data="/path/to/test.csv",project_name="my_first_project",run_name="my_first_run",)☕️ Wait for Galileo to train your model and analyze the results.✨ A link to your run will be provided automaticallyPro tip: Set your token programmatically for automated workflowsBy setting the token, you'll never be prompted to log inimportdataqualityasdqdq.config.token='MY-TOKEN'For long-lived flows like CI/CD, see our docs onenvironment variablesWhat kinds of datasets can I analyze?Currently, you can analyzeText ClassificationandNERIf you want support for other kinds,reach out!Can I use auto with other data forms?autoparamstrain_data,val_data, andtest_datacan also take as input pandas dataframes and huggingface dataframes!What if all my data is in huggingface?Use thehf_dataparam to point to a dataset in huggingfaceimportdataqualityasdqdq.auto(hf_data="rungalileo/emotion")Anything else? Can I learn more?Runhelp(dq.auto)for more information on usageCheck out ourdocsfor the inspiration behind this methodology.Can I analyze data using a custom model?Yes! Check out ourfull documentationandexample notebookson how to integrate your own model with GalileoWhat if I don't have labels to train with? Can you help with labeling?We have anapp for that! Currently text classification only, butreach outif you want a new modality!This is currently in development, and not an official part of the Galileo product, but rather an open source tool for the community.We've built a bulk-labeling tool (and hosted it on streamlit) to help you generate labels quickly using semantic embeddings and text search.For more info on how it works and how to use it, check out theopen source repo.Is there a Python API for programmatically interacting with the console?Yes! See our docs ondq.metricsto access things like overall metrics, your analyzed dataframe, and even your embeddings.ContributingRead ourcontributing doc!
Package: data-quality-tests
DATA QUALITYA library which acts as a test cases for dataframes. Simply pass in your dataframe after initial import, or at each stage of your EDA to check for data quality with one line of code.The test cases include(as of now)check for null valuescheck for duplicatescheck for dtype matchingcheck for outlierscheck for whitespaces in column headersThe test cases work as a Pass/Fail type, where Passed indicates, good data quality and Failed indicates bad data qualityExample:TEST CASE FOR NULL VALUES: Passed means that the dataframe has no null values. Failed indicates otherwise.RequirementsPython 3+PandasNumpyInstallationpipinstalldata-quality-testsUpdates & Changesthe import function changed from:fromdata_qualityimportDataQualityto the following:fromdata_quality_testsimportDataQualitynew functionoutlier_columnshas been added in this update, which displays all the columns that have outliers.For use case, refer to the get started sectiondata_quality_checknow checks for column header whitespaces for leading and trailing.new functiondtype_columnshas been added in this update, which displays all the columns that failed data type matching.For use case, refer the get started sectionGet StartedHow to use this library:Data quality checkThe most basic usage of this library, here for simplifiction,let's just se the iris dataset from seaborn library.You can use any dataset.fromdata_quality_testsimportDataQualityasdqimportseabornassns#declare any dataframedf=sns.load_dataset("iris")#pass the dataframe as belowdq.data_quality_check(df)Outlier columnsSometimes, the test case for outliers fails, this is because the dataset containes outliers.useoutier_columns(df)function to display all the columns that have outliers.NOTEIf the dataset does not have outlier columns, the output is an empty list.# display columns that have outliersdq.outlier_columns(df)Data type columnsWhen it is observed the test for dtype matching has failed, this is due to an error in the data type.
For example, sometimes numbers are in the form of strings.usedtype_columns(df)function to display all the columns that failed data type matching.#display list of columns that failed dtype matchingdq.dtype_columns(df)
Package: dataqualityx
DataQuality快速开始pipinstalldataqualityx多文件任务dataqualityx--filesC:\warehouse\user\task单文件任务dataqualityx--filesC:\warehouse\user\task\dws_test.yaml脚本编写db:type:dorishost:"127.0.0.1"port:9030user:"admin"pw:"xxx"db:"ods"vars:-var:"${dt}=$[yyyy-MM-dd-1]"tasks:-task:sql:|-- 校验 dws_test 表中前一天的数据是否存在select count(1) from dws.dws_testwhere dt = '${dt}'expected:-row:">0"alert:-task:sql:|-- 校验 dws_test 表中4种类型是否存在数据select count(1)from dws.dws_testwhere dt = '${dt}'group by typeorder by typeexpected:-row:">0"-row:">0"-row:">0"-row:">0"alert:支持命令--files /home/users/xxx: 支持指定文件路径 (必填)--alerts none: 支持全局禁止报警支持数据库mysqldorishiveclickhouseclickhousepostgresql
Package: datar
datarA Grammar of Data Manipulation in pythonDocumentation|Reference Maps|Notebook Examples|APIdataris a re-imagining of APIs for data manipulation in python with multiple backends supported. Those APIs are aligned with tidyverse packages in R as much as possible.Installationpipinstall-Udatar# install with a backendpipinstall-Udatar[pandas]# More backends support coming soonBackendsRepoBadgesdatar-numpydatar-pandasdatar-arrowExample usage# with pandas backendfromdatarimportffromdatar.dplyrimportmutate,filter_,if_elsefromdatar.tibbleimporttibble# or# from datar.all import f, mutate, filter_, if_else, tibbledf=tibble(x=range(4),# or c[:4]  (from datar.base import c)y=['zero','one','two','three'])df>>mutate(z=f.x)"""# outputx        y       z<int64> <object> <int64>0       0     zero       01       1      one       12       2      two       23       3    three       3"""df>>mutate(z=if_else(f.x>1,1,0))"""# output:x        y       z<int64> <object> <int64>0       0     zero       01       1      one       02       2      two       13       3    three       1"""df>>filter_(f.x>1)"""# output:x        y<int64> <object>0       2      two1       3    three"""df>>mutate(z=if_else(f.x>1,1,0))>>filter_(f.z==1)"""# output:x        y       z<int64> <object> <int64>0       2      two       11       3    three       1"""# works with plotnine# example grabbed from https://github.com/has2k1/plydataimportnumpyfromdatarimportffromdatar.baseimportsin,pifromdatar.tibbleimporttibblefromdatar.dplyrimportmutate,if_elsefromplotnineimportggplot,aes,geom_line,theme_classicdf=tibble(x=numpy.linspace(0,2*pi,500))(df>>mutate(y=sin(f.x),sign=if_else(f.y>=0,"positive","negative"))>>ggplot(aes(x="x",y="y"))+theme_classic()+geom_line(aes(color="sign"),size=1.2))# very easy to integrate with other libraries# for example: klibimportklibfrompipdaimportregister_verbfromdatarimportffromdatar.dataimportirisfromdatar.dplyrimportpulldist_plot=register_verb(func=klib.dist_plot)iris>>pull(f.Sepal_Length)>>dist_plot()Testimonials@coforfe:Thanks for your excellent package to port R (dplyr) flow of processing to Python. I have been using other alternatives, and yours is the one that offers the most extensive and equivalent to what is possible now withdplyr.
Package: datarade
DataradeThis library provides tools that allow datasets to be defined separately from a pipeline.OverviewThis library separates the 'how' from the 'what' when sourcing datasets and producing data pipelines. The definition of
a dataset is stored in a git repository and referenced by name in the client application. This allows the definition to
be source controlled independently from the client application. Also, by adding branch support, a dataset catalog can
hold different 'environments' as branches. This allows you to promote your dataset definition and your client
application code using your standard CI/CD process while also giving you an area to perform UAT.RequirementsPython 3.7+marshmallowpyyamlrequestsazure-devopssqlalchemypyodbcbcpInstallationThis package is hosted on PyPI:pipinstalldataradeExamplesUse datarade services to obtain metadata about your datasets from your dataset catalog:importdataraderepository_url='https://raw.githubusercontent.com/fivestack/datarade_test_catalog'dataset_catalog=datarade.get_dataset_catalog(repository=repository_url,organization='fivestack',platform='github')# no username/password since only public repos are currently supported for githubdataset=datarade.get_dataset(dataset_catalog=dataset_catalog,dataset_name='my_dataset')print(dataset.name)print(dataset.definition)Use datarade services to write datasets to a database:importdataraderepository_url='https://raw.githubusercontent.com/mikealfare/dataset_catalog_test/master'dataset_catalog=datarade.get_dataset_catalog(repository=repository_url,organization='fivestack',platform='azure-devops',username='USERNAME_TO_ACCESS_THE_GIT_REPO',password='PASSWORD_TO_ACCESS_THE_GIT_REPO')dataset_container=datarade.get_dataset_container(driver='mssql',database_name='datarade',host=r'localhost\DATARADE',username='USERNAME_TO_WRITE_TO_THE_DATABASE',password='PASSWORD_TO_WRITE_TO_THE_DATABASE')# you can do one off writes like thisdataset=datarade.get_dataset(dataset_catalog=dataset_catalog,dataset_name='my_dataset')datarade.write_dataset(dataset=dataset,dataset_container=dataset_container,username='USERNAME_TO_READ_THE_DATASET_FROM_THE_SOURCE',password='PASSWORD_TO_READ_THE_DATASET_FROM_THE_SOURCE')defwrite_dataset_wrapper(dataset_name:str,username:str=None,password:str=None):"""But it may be useful to create a function that wraps the configuration like this if you are writing several datasetsand only using one DatasetCatalog and one DatasetContainer."""dataset=datarade.get_dataset(dataset_catalog=dataset_catalog,dataset_name=dataset_name)datarade.write_dataset(dataset=dataset,dataset_container=dataset_container,username=username,password=password)write_dataset_wrapper(dataset_name='my_other_dataset',username='USERNAME_FOR_THIS_SOURCE',password='PASSWORD_FOR_THIS_SOURCE')Full DocumentationFor the full documentation, please visit:https://datarade.readthedocs.io/en/latest/
Package: dataramp
DatarampHere you will find documentation for dataramp, as well as some examples.Datarampis a Python library designed to streamline data science and data analysis workflows. It offers a collection of utility functions and tools tailored to assist data science teams in various aspects of their projects.By providing a range of functionalities, Dataramp aims to enhance productivity and efficiency in data science projects, empowering teams to focus on deriving meaningful insights from their data.Getting StartedRead the quick start guidehere.Or if you want to see some examples, you can look at the examples in theexamplesdirectory.You can install dataramp and learn more fromPyPi
Package: datar-arrow
datar-arrowThe pyarrow backend fordatar.Note that onlybaseAPIs are implemented.Installationpipinstall-Udatar-arrow# orpipinstall-Udatar[arrow]Usagefromdatar.baseimportceiling# without itceiling(1.2)# NotImplementedByCurrentBackendError# with itceiling(1.2)# 2
Package: dataray
DataRayDataRay is a class of decorators helping detect the metadata of the data.The motivation of this package is to help understand how your api request data looks like. Normally we can directly
check how the api request looks like via some UI platform such as postman etc. With this package, user may directly
get the data structure and some basic metadata of the requested dataFeaturesJson StructureFurther feature coming soonDetect the Json Structure# here is your customer request function. Now it is supposed to return list or dict you are interested in looking intofromdatarayimportdataray@dataray.raydefrequest_func(*args,**kwargs):...Then every time callingrequest_func, the json structure will be printed out.
Package: datareader
datareaderparams = datareader.get_params()
path = datareader.__path__[0]
newpath = os.path.join(path, "data/input/stanford/bookstore/video0/annotations.txt")
X = datareader.read_dataset(params, newpath)
print(X)Third partyStanford Drone Dataset annotations fromhttps://cvgl.stanford.edu/projects/uav_data/are used:Please cite also the Stanford Drone Dataset:
A. Robicquet, A. Sadeghian, A. Alahi, S. Savarese, Learning Social Etiquette: Human Trajectory Prediction In Crowded Scenes in European Conference on Computer Vision (ECCV), 2016.
Package: data-reader
Data ReaderPython package thatlazilyreads data from largeJSON,XMLandCSVfiles.Why not use the already existing libs you ask ?Well, for me, it was crucial to have a small package size that can be deployed to aServerlessenvironment as this was the main goal.This package has only one dependency, and that isijsonwhich itself is small.Installationpipinstalldata-readerAPI ReferenceMapper -data_reader.mapperReadersCSVReader -data_reader.readers.csv_readerJSONReader -data_reader.readers.json_readerXMLReader -data_reader.readers.xml_reader
Package: data-readers
DescriptionThis project is useful in any scenario where you need to read some data in a useable manner quickly and easily.It can be used to read from:csvsqlitemysqlredismongopostgresaerospikeLook athttps://pypi.python.org/pypi/data-writers/to write to them.
This library was written with the intention to be useful without data-writers but they work very well together.SetupInstallpip install data-readersUsageImport DataReaderfrom datareaders.datareader import DataReadercsvlist_of_dicts = DataReader(reader=’csv’, database=’data.csv’).read()sqlitelist_of_dicts = DataReader(reader=’sqlite’, database=’data.sqlite’, table=’DataTable’).read()mysqllist_of_dicts = DataReader(reader=’mysql’, database=’data’, user=’root’, table=’DataTable’).read()redislist_of_dicts = DataReader(reader=’redis’, database=’1’).read()mongolist_of_dicts = DataReader(reader=’mongo’, database=’data’, table=’DataTable’).read()postgreslist_of_dicts = DataReader(reader=’postgres’, database=’data’, table=’DataTable’).read()aerospikelist_of_dicts = DataReader(reader=’aerospike’, namespace=’data’, set=’DataTable’).read()
Package: datareadingrequests
Data Reading RequestsA new client for Energize Andover's Building Energy Gateway, with a focus on clarity and usability.FeaturesIt's packaged, so you can install it with pip!It's fully unit-tested.It's open source,
and contributions are welcome.API Changes from building_data_requestsFor compatibility, datareadingrequests' function definitions are similar to those of building_data_requests.
However, there are a few key differences between the two modules:Instead of a tuple, datareadingrequests'get_value()returns a DataReading namedtuple.
This allows you to use the original tuple notation or the cleaner dot notation.
Read more about namedtupleshere.datareadingrequests has a predictable, single return type forget_value().
With building_data_requests,get_value()could return a valid result orNone.
Here, it can only return a valid result; it raises an exception if the server returns no data.
The reasoning for this is well-explained by williballenthin:I've learned thatreturning more than one type of data from a function is a recipe for trouble.
For example, when a function can return a stringora list,
then every place that the function is called must check "is it a string or a list?".
If the programmer forgets this, then inevitably,
the program breaks at an inconvenient time.
By extension, if a function returns a string orNone,
then every invocation must check "is the resultNone?".
This is easy to forget, and leads to latent bugs.
With the existing style, forgetting atry/exceptblock is also a bug,
but when the exception is generated,
the programmer gets a very explicit stack trace with easy-to-find line number.In the same way, datareadingrequests'get_bulk()raises an exception
if the server returns no data for any specific instance.datareadingrequests has no way (currently) to change hostname or port.datareadingrequests does not retry requests without SSL.SetupUse your favorite Python package manager, and do as you would with pandas, matplotlib, etc.pip:pip install datareadingrequests
pip freeze > requirements.txtPoetry:poetry add datareadingrequests
Package: datarecipe
# datarecipe## Introduction`datarecipe` is an open-source package designed for data practitioners. It aims to provide comprehensive, robust, and user-friendly data tools.## Dependencies- `sqlalchemy`: <2.0## Functions### chat()#### PurposeThe `chat()` function allows you to pass in a small dataset and a user question. It then returns an answer from ChatGPT-4 based on the data and the cost of generating this answer. Additionally, the function creates a YAML file in the working directory, recording the date of the query, the question, and the cost.#### Parameters- `df`: The small dataset you want to analyze (Type: DataFrame).- `question`: The question we want answered (Type: String).- `key`: Your ChatGPT API key (Type: String).- `yaml_name`: The file name of the yaml file (Type: String).- `system`: The messages describe the behavior of the AI assistant (Type: String).- `cost`: Whether to return the problem cost, True returns the problem cost, False returns the problem token number, default is True. (True/False).- `daily_cost_limit`: Single-day question spending limit, default is 10 (Type: Number).#### Return Values- ChatGPT response (Type: String)- question cost, in dollars (Type: Number).#### Example```python# Initialize your questionuser_question = "What is the average value in Column1?"# Call the chat() functionresponse, cost = datarecipe.chat(df=df, question=user_question, key=your_api_key)# Output the response and costprint(response)print(cost)```---### send_email()#### PurposeThe `chat()` function allows you to pass in a small dataset and a user question. It then returns an answer from ChatGPT-4 based on the data and the cost of generating this answer. Additionally, the function creates a YAML file in the working directory, recording the date of the query, the question, and the cost.The `send_email()` function can be used in scripts to send emails to specified addresses for various purposes such as exception reminders or periodic script reports.#### Parameters- `subject`: Your e-mail subject (Type: str).- `body`: Your e-mail content (Type: str).- `send_email_address`: E-mail address of the sender (Type: str).- `send_email_password`: E-mail password of the sender (Type: str).- `receive_email_address`: E-mail address of the receiver (Type: str).- `smtp_address`: SMTP address of sender's e-mail (Type: str).- `smtp_port`: SMTP port of sender's e-mail (Type: int).#### Return Values- None#### Example```pythondatarecipe.send_email(subject='your_email_title',body='your_email_content',send_emial_address='sender_emial_address',send_emial_password='sender_emial_password',receive_email_address='your_email_address')```