Saravanan666/pypi-org huggingface dataset:
Package: dataito
DataITO安装(install)安装开发版（Install development version）py-mpipinstall--index-urlhttps://test.pypi.org/simple/--no-depsdataito安装稳定版（ Install stable version ）pipinstalldataito使用手册 (中文版)Python数据输入(Input)、转换(transform)、输出(output)，一行代码读取/转换多种格式的数据文件dataito仅有三个函数，分别是read()、transform()、save()，具体参数及调用方式如下：格式目前支持的读取格式txtxlsxcsvjson（仅支持结构化数据）目前支持的转换格式dataframe (pandas)array (numpy)list目前支持的保存格式xlsx（目前仅支持保存为xlsx，在考虑是否要增加自定义格式保存功能）调用方式read( )read(filepath)注：只能读取支持的文件格式（建议filepath之前加个r，具体看example）transform( )transform(data,'parameter')parameter中填写为需要转换的目标数据类型，其与type(data)获取的数据类型的关系如下：typetype(data)dataframe/pandas<class 'pandas.core.frame.DataFrame'>array/numpy<class 'numpy.ndarray'>list<class 'list'>>>>data=dataito.transform(data,'dataframe')>>>type(data)<class'pandas.core.frame.DataFrame'>>>>data=dataito.transform(data,'array')>>>type(data)<class'numpy.ndarray'>>>>data=dataito.transform(data,'list')>>>type(data)<class'list'>save( )save(filepath)（建议filepath之前加个r，具体看example）exampleimportdataitofilepath=r'data/data.xlsx'#读取支持格式的数据文件data=dataito.read(filepath)#调用函数读取(读取其他支持的格式也是这个函数)data=dataito.transform(data,'dataframe')#数据格式转换为想要的格式（转换为其他支持的格式也是这个）dataito.save(data,r'D:\data\data.xlsx')#保存在data文件夹（默认文件名为data）User manual (English version)formatPython data input (i), transform (t), output (o), a line of code to read / convert a variety of formats of data filesCurrently supported read formatstxtxlsxcsvjson (only supports structured data)Currently supported conversion formatsdataframearray (numpy)listCurrently supported save formatsxlsx ( it only supports saving as xlsx. We are considering whether to add the function of saving in custom format.)Call moderead( )read(filepath)Note: only the supported file formats can be read (it is recommended to addrbefore filepath, see example for details)transform( )transform(data,'parameter')parameter is the target data type to be converted, and its relationship with the data type obtained by type (data) is as follows:typetype(data)dataframe/pandas<class 'pandas.core.frame.DataFrame'>array/numpy<class 'numpy.ndarray'>list<class 'list'>>>>data=dataito.transform(data,'dataframe')>>>type(data)<class'pandas.core.frame.DataFrame'>>>>data=dataito.transform(data,'array')>>>type(data)<class'numpy.ndarray'>>>>data=dataito.transform(data,'list')>>>type(data)<class'list'>save( )save(filepath)(it is recommended to addrbefore filepath, see example for details)exampleimportdataitofilepath=r'data/data.xlsx'#Read data files in supported formatsdata=dataito.read(filepath)#Call the function to read (read other supported formats as well as this function)data=dataito.transform(data,'dataframe')#Convert the data format to the desired format (and other supported formats)dataito.save(data,r'D:\data\data.xlsx')#Save in the data folder (the default file name is data). If the path is not written, the file is saved in the root directory
Package: datajazz
DataJazzTools to jazz up your data.DataJazz is toolkit for manipulating and optimizing your data for analysis, machine learning, extraction, transformation, and loading (ETL).ContributingDataJazz is an open-source project founded and maintained to better serve the data science and machine learning community. Please feel free to submit pull requests to contribute to the project. By participating, you are expected to adhere to DataJazz'scode of conduct.Installationpip install datajazzExample usageCreate a dataframe with different datatypesimport pandas as pd
import numpy as np

rng = pd.date_range('2015-02-24', periods=5, freq='20H')
df = pd.DataFrame({ 'Start_Time': rng, 'Values': np.random.randn(len(rng)), 'Categories': ['A']*len(rng) }) 
df.head()Create time-of-time featuresimport datajazz as dj
df = dj.timeoftime(df)One-hot encode your categorical columnsimport datajazz as dj
df = dj.onehot_categories(df)Remove redundant rows and columnsimport datajazz as dj
df = dj.remove_redundancies(df)Many more use cases to come! Submit a pull request to add a new use case.
Package: datajet
DataJetA Data Dependency Graph Framework and ExecutorDataJet abstracts over function calls by mapping inputs through a graph of functions to desired outputs. As a programmer, you declare your data transformations (functions of inputs to outputs) once, and datajet will handle mappinganyinput toanyoutput reachable by the graph of functions.Key FeaturesLazy: Only Evaluate and return the data you needDeclarative: Declare Data and functions on the data explicitly, using plain pythonDependency-Free: Just Python.InstallationRequirements:Python >=3.8To Get Started, Install DataJet From pypi:pipinstalldatajetWhy would I use this?DataJet simplifies the codebase of dynamic systems with mutliple ways to calculate a datapoints from different inputs.DataJet de-couples downstream calculations from the mechanics of calculating upstream dependencies.Quickstartfromdatajetimportexecutedollars=[7.98,20.94,37.9,30.31]units=[1,3,5,4,]defprices(dollars,units):return[d/uford,uinzip(dollars,units)]defaverage_price(prices):returnsum(prices)/len(prices)defaverage_price_rounded_down(average_price):returnaverage_price*1000//10/100datajet_map={"prices":prices,"average_price":average_price,"average_price_rounded_down":average_price_rounded_down,}execute(datajet_map,context={"dollars":dollars,"units":units,},fields=['average_price_rounded_down']){'average_price_rounded_down':7.52}And, if you have prices, you can directly get what you need:prices=[3.99,4.49,2.89,2.79,2.99]execute(datajet_map,context={"prices":prices,},fields=['average_price','average_price_rounded_down']){'average_price':3.4299999999999997,'average_price_rounded_down':3.42}Important DetailsKeys can be any hashable. The value corresponding to each key can be a function or an object. The functions can have 0 or more parameters. The parameter names must correspond to other keys in the dict if no explicitly defined inputs to the callable are declared in the map. SeeDatamap referencefor how to explicitly define inputs.You can also define multiple ways of calculating a piece of data via defining a list of functions as the value to the key. Again, each function's parameters must correspond to other keys in the dict, or else you can define which other keys should be inputs to the function via explicitly defining inputs.Full Documentationhttps://bmritz.github.io/datajet/DevelopmentTo create the development environment locally:git clone
make installThis will start apoetry shellthat has all the dev dependencies installed. You can rundeactivateto exit the shell.To run testsmake testDevelopment troubleshootingIf you see:urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>Go to /Applications/Python3.x and run 'Install Certificates.command'Built on ideas inspired bywilkerlucio/Pathom3stitchfix/hamilton
Package: datajoint-babel
datajoint-babelGenerate schema code from model definitions for both Python and MATLAB (and eventually vice versa).Say you're a lab that uses both Python and MATLAB, this lets you declare your models once and then generate
both Python and MATLAB versions of them, rather than having two potentially mutually contradictory sets of
models. Keep explicit structure and avoid implicit model recreation from the database <3.More generally a pythonic adapter interface from an explicit data model (thankspydantic!) to datajoint models so other tools can
patch in more easily!So far just a single afternoon project, but will be the means by which autopilot interfaces directly with datajoint :)ExampleSource a model from a string>>>fromdatajoint_babel.modelimportTable>>>frompprintimportpprint>>>tab=Table.from_definition(name='User',tier='Manual',definition="""# database usersusername : varchar(20)   # unique user name---first_name : varchar(30)last_name  : varchar(30)role : enum('admin', 'contributor', 'viewer')""")>>>tab.dict(){'name':'User','tier':'Manual','comment':{'comment':'database users'},'keys':[{'name':'username','datatype':{'datatype':'varchar','args':20,'unsigned':False},'comment':'unique user name','default':None}],'attributes':[{'name':'first_name','datatype':{'datatype':'varchar','args':30,'unsigned':False},'comment':'','default':None},{'name':'last_name','datatype':{'datatype':'varchar','args':30,'unsigned':False},'comment':'','default':None},{'name':'role','datatype':{'datatype':'enum','args':["'admin'"," 'contributor'"," 'viewer'"],'unsigned':False},'comment':'','default':None}]}>>>pprint(tab.__dict__){'attributes':[Attribute(name='first_name',datatype=DJ_Type(datatype='varchar',args=30,unsigned=False),comment='',default=None),Attribute(name='last_name',datatype=DJ_Type(datatype='varchar',args=30,unsigned=False),comment='',default=None),Attribute(name='role',datatype=DJ_Type(datatype='enum',args=["'admin'"," 'contributor'"," 'viewer'"],unsigned=False),comment='',default=None)],'comment':Comment(comment='database users'),'keys':[Attribute(name='username',datatype=DJ_Type(datatype='varchar',args=20,unsigned=False),comment='unique user name',default=None)],'name':'User','tier':'Manual'}Export to python...>>>print(tab.make(lang='python'))@schemaclassUser(dj.Manual):definition="""# database usersusername : varchar(20) # unique user name---first_name : varchar(30)last_name : varchar(30)role : enum('admin', 'contributor', 'viewer')And to MATLAB>>>print(tab.make(lang='matlab'))%{# # database users# username : varchar(20) # unique user name---# first_name : varchar(30)# last_name : varchar(30)# role : enum('admin', 'contributor', 'viewer')%}classdefUser<dj.Manualend
Package: datajoint-connection-hub
DataJoint Hub PluginThis is an official DataJoint Python plugin for connection to DJNeuro's hosted instances.https://djneuro.io/servicesNote: This plugin adds features to thedatajointpackage and thus requires it. Users should not import directly from the plugin but instead simplyimport datajointto access plugin features.Run tests locally$  ./serve.sh up LOCAL
$$ docker exec -it datajoint_connection_hub-python_app_1 sh
$$ nosetests -v --tests=/home/dja/datajoint-python/tests
$$ nosetests -vw tests
$$ exit
$  ./serve.sh down LOCALNote: Make sure to have a valid Git SSH key to access necessary private repos.
Package: datajoint-link
:link: datajoint-linkA tool for convenient and integrity-preserving data sharing between database servers.:floppy_disk: InstallationOnly users interacting with the destination of the data need to install the datajoint-link package:pipinstalldatajoint-link:wrench: SetupSourceDatajoint-link requires access to the database server from which data will be pulled. It is recommended to create a new user for this purpose:CREATEUSER'djlink'@'%'IDENTIFIEDBY'secret-password';The user needs to have certain privileges on the table from which data will be pulled:GRANTSELECT,REFERENCESON`source\_schema`.`source\_table`TO'djlink'@'%';Each table from which data will be pulled also needs an additional helper table:GRANTALLPRIVILEGESON`helper\_schema`.`helper\_table`TO'djlink'@'%';In order to preserve data integrity across the link regular users must not have any privileges on this helper table.DestinationDatajoint-link needs to be configured with the username and password of the user created in the previous section. This is accomplished via environment variables:LINK_USER=djlinkLINK_PASS=secret-password:computer: UsageThe destination table is created by passing information about where to find the source table to thelinkdecorator:fromlinkimportlink@link("databaseserver.com","source_schema","helper_schema","helper_table","destination_schema")classTable:"""Some table present in the source schema on the source database server."""Note that the name of the declared class must match the name of the table from which the data will be pulled.The class returned by the decorator behaves like a regular table with some added functionality. For one it allows the browsing of rows present in the source:Table().sourceAll the rows can be pulled like so:Table().source.pull()# Hint: Pass display_progress=True to get a progress barThat said usually we only want to pull rows that match a certain criteria:(Table().source&"foo = 1").pull()The deletion of already pulled rows works the same as for any other table:(Table()&"foo = 1").delete()The deletion of certain rows from the destination can also be requested by flagging them in the corresponding helper table:row=(Helper()&"foo = 1").fetch1()(Helper()&row).delete()row["is_flagged"]="TRUE"Helper().insert1(row)Theflaggedattribute makes the deletion of flagged rows from the destination table convenient:(Table()&Table().source.flagged).delete()Deleting a flagged row automatically updates its corresponding row in the helper table:assert(Helper()&"foo = 1").fetch1("is_deprecated")=="TRUE"# No error!Now it is save to delete the row from the source table as well!:package: External StorageData stored in a source table that refers to one (or more) external stores can be stored in different store(s) after pulling:@link(...,stores={"source_store":"destination_store"})classTable:...Note that all stores mentioned in the dictionary need to be configured viadj.config.:white_check_mark: TestsClone this repository and run the following command from within the cloned repository to run all tests:dockercomposerunfunctional_teststests
Package: datajudge
datajudgeExpress and test specifications against data from database.DocumentationUsagedatajudgecan either be installed via pypi withpip install datajudgeor via conda-forge withconda install datajudge -c conda-forge.Please refer to theGetting Startedsection of our documentation for details.Expressing an expectations between different tables from a database may look as such:fromdatajudgeimportBetweenRequirementcompanies_between_req=BetweenRequirement.from_tables(db_name1="example",table_name1="companies",db_name2="example",table_name2="companies_archive",)companies_between_req.add_row_superset_constraint(columns1=["name"],columns2=["name"],constant_max_missing_fraction=0)
Package: datajudge-client
DataJudge Python ClientInteracts withDataJudge.Installation$pipinstalldatajudge-clientUsage$python-mdatajudge-client--keyYOUR_API_KEYteamname
$python-mdatajudge-client--keyYOUR_API_KEYleaderboard
$python-mdatajudge-client--keyYOUR_API_KEYsubmit--inputSUBMISSION_FILE
Package: datajunction-reflection
DJ Reflection ServiceThe reflection service polls the DJ core service for all nodes with associated tables, whether source
tables or materialized tables. For each node, it refreshes the node's schema based on the associated
table's schema that it retrieves from the query service. It also retrieves the available partitions and
the valid through timestamp of these tables and reflects them accordingly to DJ core.This service uses a celery beat scheduler, with a configurable polling interval that defaults to once per
hour and async tasks for each node's reflection.
Package: datajunction-server
DataJunctionIntroductionDataJunction (DJ) is an open sourcemetrics platformthat allows users to define
metrics and the data models behind them usingSQL, serving as asemantic layeron top of a physical data warehouse. By leveraging this metadata, DJ can enable efficient
retrieval of metrics data across different dimensions and filters.Getting StartedTo launch the DataJunction UI with a minimal DataJunction backend, start the default docker compose environment.dockercomposeupIf you'd like to launch the full suite of services, including open-source implementations of the DataJunction query service and
DataJunction reflection service specifications, use thedemoprofile.dockercompose--profiledemoupDJUI:http://localhost:3000/DJ Swagger Docs:http://localhost:8000/docsDJQS Swagger Docs:http://localhost:8001/docsJaeger UI:http://localhost:16686/searchJupyter Lab:http://localhost:8888How does this work?At its core, DJ stores metrics and their upstream abstractions as interconnected nodes.
These nodes can represent a variety of elements, such as tables in a data warehouse
(source nodes), SQL transformation logic (transform nodes), dimensions logic,
metrics logic, and even selections of metrics, dimensions, and filters (cube nodes).By parsing each node's SQL into an AST and through dimensional links between columns,
DJ can infer a graph of dependencies between nodes, which allows it to find the
appropriate join paths between nodes to generate queries for metrics.
Package: datakeeper
[![pipeline status](https://gitlab.com/crosenbrock/app-datakeeper/badges/master/pipeline.svg)](https://gitlab.com/crosenbrock/app-datakeeper/commits/master)
[![coverage report](https://gitlab.com/crosenbrock/app-datakeeper/badges/master/coverage.svg)](https://gitlab.com/crosenbrock/app-datakeeper/commits/master)# Data Keeper Templated SystemThis is a python-based project that generates other repositories for generalized
data privacy and consent systems.
Package: data-kennel
data_kennelData Kennel is a CLI tool for managing Datadog infrastructure.About AmplifyAmplify builds innovative and compelling digital educational products
that empower teachers and students across the country. We have a long
history as the leading innovator in K-12 education - and have been
described as the best tech company in education and the best education
company in tech. While others try to shrink the learning experience into
the technology, we use technology to expand what is possible in real
classrooms with real students and teachers.Learn more athttps://www.amplify.comGetting StartedPrerequisitesData Kennel requires the following software to be installed: * python
>= 2.7For development: * tox >= 2.9.1Installing/BuildingData Kennel can be installed from pip.pip install data_kennelFor local development, Data Kennel also includes a setup script.python setup.py developCredentialsData Kennel expects your Datadog API and APP keys to be available as
environment variables, asDATADOG_API_KEYandDATA_KENNEL_APP_KEY. Here is an example:# Data Kennel EnvvarsexportDATADOG_API_KEY="change_me"exportDATA_KENNEL_APP_KEY="change_me"You can create API and APP keys in theDatadog
console.Running TestsData Kennel has lint checks and unit tests for use when developing.
Simply run tox.toxSupported OperationsMonitor ManagementData Kennel currently supports listing, syncing, and deleting simple and
composite monitors. Composite monitors are monitors that are composed of
several other monitors. This is achieved through thedk_monitorcommand.dk_monitorhas the following options available: * list *
update * deleteSeedk_monitor-hfor more information anddata_kennel.yml.examplefor an example of the configuration file.RoadmapAdd support forORboolean operator in composite monitors.Add support for managing Datadog dashboards.Add support for managing Datadog downtimes.Add support for sending arbitrary Datadog events.Add support for queryng arbitrary Datadog metrics.Responsible DisclosureIf you have any security issue to report, contact project maintainers
privately. You can reach us atgithub@amplify.comContributingWe welcome pull requests! For your pull request to be accepted smoothly,
we suggest that you: 1. For any sizable change, first open a GitHub
issue to discuss your idea. 2. Create a pull request. Explain why you
want to make the change and what it’s for. We’ll try to answer any PRs
promptly.
Package: datakick
datakick is a python wrapper for theDatakickopen product database API.UsageInstallation:python setup.py installorpip install datakickSample Code:First import the module:>>>importdatakickSearch for products by barcode:>>>gtin14="037000062219">>>product=datakick.find_product(gtin14)>>>print("{}-{}".format(product.brand_name,product.name))'Crest Pro-Health Clean Mint Toothpaste'Search for products with a key:>>>products=datakick.search("Toothpaste")>>>forproductinproducts:...print("{}-{}".format(product.brand_name,product.name))'Crest Pro-Health Clean Mint Toothpaste''Sensodyne Fresh Impact Toothpaste'# etc.Add/modify products in the database:>>>gtin14="011110491503">>>product=datakick.add_product(...gtin14,brand_name="Big K",name="Diet Cola",size="355mL"...)>>>print("{}-{}-{}".format(...product.brand_name,product.name,product.size)...)'Big K Diet Cola 355mL'Add images to products in the database:>>>gtin14="011110491503">>>img_path="path/to/your/image/cola.jpg"# only .jpg or .jpeg allowed!>>>img_url=datakick.add_image(gtin14,img_path)>>>print(img_url)'https://d2b9vdin3yve6y.cloudfront.net/1a888191-e530-4d55-a871-00a0994d75c0.jpg'List the products (on a page):>>>products=datakick.list_products(5)# each page returns 100 products>>>forproductinproducts:...print(product.gtin14)'016000437692''016000439894'# etc.Optional Parameters for Adding/Modifying a product:Optional ParametersTypeUnitsExamplenamestringn/a“Toothpaste”brand_namestringn/a“Colgate”sizestringn/a“20oz”ingredientsstringn/a“Milk, Chocolate, Sugar”serving_sizestringn/a“2 tbsp.”servings_per_containerstringn/a“2 cookies”caloriesintn/a200fat_caloriesintn/a100fatint/floatgrams10saturated_fatint/floatgrams10trans_fatint/floatgrams0polyunsaturated_fatint/floatgrams5monounsaturated_fatint/floatgrams5cholesterolintmilligrams20sodiumintmilligrams40potassiumintmilligrams60carbohydrateintgrams20fiberintgrams10sugarsintgrams6proteinintgrams4authorstringn/a“First M. Last”publisherstringn/a“MyPublisher”pagesintn/a400alcohol_by_volumeint/floatpercent20Exceptions:datakick.exceptions.ImageTooLarge- Will be thrown if the image provided to datakick.add_image is too large (>1MB)datakick.exceptions.InvalidImageFormat- Will be thrown if the image provided to datakick.add_image is of the wrong file format (only .jpg or .jpeg allowed).requests.exceptions.HTTPError- Will be thrown if the gtin14 provided is invalid or not found in the product database.
Package: datakit-dworld
Commands to manage project integration with data.world.Free software: ISC licenseFeaturesCreate a project (dataset) on data.worldPush data files from a local directory to a data.world projectSetup instructionsAssuming you haveDataKitinstalled, run the following to install thedatakit-dworldplugin:$ pip install -e 'git+https://github.com/associatedpress/datakit-dworld.git#egg=datakit-dworld'Create a configuration file at~/.datakit/plugins/datakit-dworld/config.jsonto tell the plugin about your data.world account. It should look like this:{
  "username": "USERNAME",
  "api_token": "API_TOKEN"
}ReplaceUSERNAMEwith the username you will use on data.world to publish
datasets.Log in to data.worldas that same user, and go to yourAdvanced settingspage. You’ll see two long tokens under the “API Token” section. Copy the one
labeled “Admin”, and paste that into your configuration file in place ofAPI_TOKEN.How to useWhen in a datakit project, you’ll have two new commands:datakit dworld create, which creates a new project (dataset) on
data.world. This requires a--slugoption to help set the project’s
URL. For example, if your username isexample-human, then runningdatakit dworld create--slugexample-datasetwill create an empty
dataset athttps://data.world/example-human/example-dataset.datakit dworld push, which uploads data files (CSV, etc.) from your
project to data.world. This defaults to uploading any files in your
project’sdata/publicdirectory, but you can specify a different
directory with--source-dir.datakit dworld summary, which updates the dataset’s overall Markdown
summary on data.world. This assumes a Markdown file exists atpublish/distro_summary.md; if that is not present, it creates one based
on the dataset’s existing summary.CreditsThis plugin was created withCookiecutterand theassociatedpress/cookiecutter-datakit-pluginproject template (a modified version of the most excellentaudreyr/cookiecutter-pypackage).