Saravanan666/pypi-org huggingface dataset:
Package: datacycle
DatacycleGetting startedcp .env.example .env
vim .env
source .env

poetry install --extras all
poetry run datacycledocker build -f Dockerfile -t datacycle .
docker run -it --rm --env-file .env datacycleMac requirementsbrew install mongodb/brew/mongodb-database-tools
brew install libpq
brew link --force libpq
npm install elasticdump -gLinux requirementsapt install -y mongo-tools
apt install -y postgresql-client
npm install elasticdump -gHow todatacycle --help
datacycle doctor

datacycle mongo "mongodb://user:password@localhost:27017/test1?authSource=admin" "mongodb://user:password@localhost:27017/test2?authSource=admin" --transform "
    transforms {
        test1 {
            before-transform {}
        }
    }
"

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot --transform ops.hocon

datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin gs://datacycle-test/test1/snapshot
datacycle mongo mongodb://user:password@localhost:27017/test1?authSource=admin test1/snapshot

datacycle mongo gs://datacycle-test/test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo gs://datacycle-test/test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo gs://datacycle-test/test1/snapshot test2/snapshot

datacycle mongo test1/snapshot mongodb://user:password@localhost:27017/test2?authSource=admin
datacycle mongo test1/snapshot gs://datacycle-test/test2/snapshot
datacycle mongo test1/snapshot test2/snapshotProvidersPostgreshttps://www.postgresql.org/docs/9.1/backup.htmlSQL dumpfile system snapshotcontinuous archivingpg_dump --clean "postgres://user:password@localhost:5432/test" | gzip > dump.gz
gunzip -c dump.gz | psql "postgres://user:password@localhost:5432/test"Mongohttps://docs.mongodb.com/manual/core/backups/BSON dumpfile system snapshotCDCmongodump --uri="mongodb://user:password@localhost:27017/test?authSource=admin" --out=dump --numParallelCollections=10 -v --gzip
mongorestore --uri="mongodb://user:password@localhost:27017/test?authSource=admin" dump/test --numParallelCollections=10 -v --gzipElasticsearchhttps://github.com/elasticsearch-dump/elasticsearch-dumpdumpelasticdump --input=https://localhost:9200 --output=$ --limit 2000 | gzip > dump.gz
Package: data-dag
Overviewdata-dagis a library for writing data-defined Airflow DAGs and operators.Installationpip install data-dagExampleRe-usable operator and DAG template can be stored in a central location, such as a custom Airflow plugin (or a package withindags/works fine too):# plugins/my_factories/download.pyfromdata_dag.operator_factoryimportOperatorFactoryfromdata_dag.dag_factoryimportDagFactoryfromurllib.requestimporturlretrievefromtypingimportListtry:fromairflow.operators.emptyimportEmptyOperatorexceptImportError:fromairflow.operators.dummyimportDummyOperatorasEmptyOperatorfromairflow.providers.http.sensors.httpimportHttpSensorfromairflow.operators.pythonimportPythonOperatorfromairflow.utils.task_groupimportTaskGroupclassDownloadOperator(OperatorFactory):"""An operator factory for safely downloading files to a known location"""name:strurl:strpath:strdefmake_operator(self):withTaskGroup(group_id=f'download_{self.name}')asgroup:check=HttpSensor(task_id='check_exists',endpoint=self.url)download=PythonOperator(task_id=f'download',python_callable=lambda:urlretrieve(self.url,self.path))check>>downloadreturngroupclassDownloaderDag(DagFactory):"""A DAG factory for producing simple DAGs that just download a bunch of files"""downloads:List[DownloadOperator]def_make_dag(self):start=EmptyOperator(task_id='start')end=EmptyOperator(task_id='end')fordownloadinself.downloads:start>>download.make_operator()>>endThen a definition for a particular DAG can live in a data file:# dags/yaml/sample_dag.yamldag_id:sample_dagdescription:An example of how to write a data-driven DAGschedule:'@daily'start_date:'2020-01-01T00:00:00'downloads:-name:dataurl:https://www.example.com/data.zippath:data.zip-name:manifesturl:https://www.example.com/manifest.jsonpath:manifest.jsonThat data file can then be loaded into a DAG. Per Airflow's requirements, this must be done in a file located indags/and the result must be saved into a uniquely named global variable. The simplest possible example is this:# dags/sample_dag.pyfromyamlimportsafe_loadfrommy_factories.downloadimportDownloaderDagwithopen('yaml/sample_dag.yaml','r')asf:dag_data=safe_load(f)dag=DownloaderDag.model_validate(dag_data).make_dag()Multiple DAGsObviously, using a template isn't much use if you only fill it in once. Here's a simple example of a loader that will load any number of YML files from a folder and publish each one as a DAG in Airflow:# dags/load_yml_files.pyfrompathlibimportPathfromairflowimportDAGimportyamlfrommy_factoriesimportBaseDagdag_dir=Path(__file__).parent# For each YAML file in a particular directory...foryaml_file_pathindag_dir.glob('typical_dags/**.yml'):withopen(yaml_file_path)asf:dag_metadata=yaml.safe_load(f)# ... generate a DAG from that metadatadag_metadata_obj=BaseDag.model_validate(dag_metadata)dag=dag_metadata_obj.make_dag()# See https://www.astronomer.io/guides/dynamically-generating-dags/dag_name=yaml_file_path.with_suffix('').nameglobals()[dag_name]=dagDocumentationComplete docs can be found here//: # (- Add DAG loader factory to enable a 2-liner DAG python file//: # (- Add dependency group mixin or something )
Package: data-dashboard
data-dashboardShort descriptionCreates a simple static HTML dashboard with providedX,ydata to help
users see what's going on in their data, assists in making decisions regarding
features and finds the best "baseline" Machine Learning Model.Longer DescriptionInstallation and Usagedata_dashboardlibrary allows you to build HTML Dashboard visualizing not only the data and relationships between
features but also automatically search for the best 'baseline' sklearn compatible Model.You can install the package via pip:pip install data-dashboardTo create a Dashboard you need the data:X,yand theoutput_directorywhere the HTML files will be placed.
You can use toy datasets fromexamples(e.g.irisdataset) included in the library as well:fromdata_dashboardimportDashboardfromdata_dashboard.examplesimportirisoutput_directory="your_path/dashboard_output"X,y,descriptions=iris()# descriptions is additional argument described further in docsdsh=Dashboard(X,y,output_directory,descriptions)dsh.create_dashboard()Created HTML Dashboard will contain 3 subpages for you to look:Overview with summary statistics of the dataFeatures where you can dig deeper into each feature in the dataModels showing search results and models performancesDescriptiondata_dashboardaims to help you in those initial moments when you get your hands on new data and you ask yourself
a question "Now what?".Instead of going through Jupyter Notebook tables and visualizations,data_dashboardgathers all that information in one place in a user-friendly interface. What is more, automated 'baseline' sklearn Model
is created - you can then adjust not only parameters and algorithms, but also see how your changes affect the
performance.There is also an educational twist todata_dashboard, as explanations on correlations and transformations might
alleviate pains that beginner Data Scientists might encounter (e.g. what do Transformers really do with my features?).
Furthermore, provided visualizations might also assist in manual feature engineering.Last but not least,data_dashboardtries to put more emphasis on the design of both HTML and Visualization
elements. If you are not the biggest fan of default matplotlib plots, then you can givedata_dashboarda try -
perhaps the styling will suit your taste!Working ExampleExample with Titanic dataset can be found here:https://example-data-dashboard.herokuapp.com/DocumentationDocumentation can be found here:https://data-dashboard.readthedocs.io/AuthorMIT License Copyright (c) 2021 Maciej DowgirdFor contact use:dowgird.maciej@gmail.com
Package: data-dealer
# data-dealer (0.0.4)
A Python wrapper for data virtualization, ETL and serialization as a pandas dataframe.## Installation` pip installdata-dealer`## Integrations
- .json and .csv file formats
- MS SQL Server
- AWS DynamoDB
- AWS Redshift
- AWS Athena (In Development)
- AWS s3 (In Development)## Registry
In order to begin using the data dealer, you must register some suppliers. The following CLI commands will help you manage your registry#### Register a new supplier` dealer registry-stest-supplier-p"dbms=mssql;host=localhost;uname=MyUsername;pword=MyPassword" add`Valid supplier properties:
- dbms (athena, mssql, dynamo)
- host
- port
- uname
- pword
- aws_key
- aws_secret#### Load a registry from file` dealer registry-s~/registry.yaml load `#### Display current registry` dealer registry [display|show] `#### Remove a supplier from the registry` dealer registry-stest-supplierremove `## To-Do List:
Too many things to count
Package: datadeck
DescriptionDataDeck (once known as datapkggui) is a graphical frontend for the dpm tool (once known as datapkg)
(http://dpm.readthedocs.org/en/latest/index.html).
It is aiming to become the standard CKAN (http://thedatahub.org/) graphical tool for searching, fetching
and distributing open data sets.It is a 10 weeks project started on 2011-09-26 and developed by
Daniel Graziotin <d@danielgraziotin.it> (http://task3.cc) during a stage
performed at the Fondazione Bruno Kessler (http://fbk.eu).Please take a look at docs/report.{odt, pdf} for many other info.WebsiteThe official website of the project ishttp://task3.cc/projects/datadeck.
The source code is hosted on GitHub (http://github.com/dgraziotin/datadeck)VersionsDataDeck uses Semantic Versioning for defining releases. (http://semver.org)ChangelogA sort of changelog can be retrieved athttp://task3.cc/tag/datadeck-changelog/Only some relases are posted in the website.
For in-depth changelog, please refer to GitHub project page.Compatibility NotesUp to v0.2.3-datapkgguilast, the project was called datapkggui and supported datapkg 0.8.
From v0.2.4, the project is renamed to DataDeck and supports dpm >= 0.10.IMPORTANTDataDeck is in development while dpm is also under heavy development. Please note that dpm may have not been
“officially” released when you try DataDeck. Therefore, dpm has been pulled out from the dependency check.
Please be sure to obtain dpm athttps://github.com/okfn/dpmif it has not already been published elsewhere.
Package: datadelve
DataDelveWorking with complex nested data can be tedious. If you have to access any objects that are four layers deep in a JSON response from a web service, you quickly tire of writing square brackets.
Much better would be to have a simple way of accessing data through a simple syntax.jsonpointeris a perfect match, it looks just like paths through a filesystem.
Applying this information to the data structures makes it easy and convenient.UsagefromdatadelveimportDataDelverdata=["your annoying data here"]delver=DataDelver(data)element=delver.get("/dict/keys/and/1/list/index")subset=delver.cd("/particular/key/to/focus/on")delver.set("/path/to/change","new")delver.delete("/bad")SupportThis package grew around a series of JSON files, so that is the primary focus. It therefore
expects data structures with dicts and lists. As the implementation is turned over to
jsonpointer, it will work for anything that implements__getitem__(str)or that is registered
as acollections.abc.Sequenceand implements__getitem__(int).Raw data, loaded from whatever source, is accepted. So are JSON files. These have the added
benefit of being flyweight instances, so all views on the same file reference the same object.
This way none of them can get out of sync and make writes indeterminate as to what updates have
actually been made. As YAML isn't in the standard library, I've split support for that into a
separate project:datadelve_yaml. YAML files are
treated the same as JSON.
Package: data-detective-airflow
Data Detective AirflowData Detective Airflowis a framework
whose main idea is to add the concepts of work and result toApache Airflow.
It supports tasks testing overApache Airflow.
Data Detective Airflow allows developers to create workflows in the form of directed acyclic graphs (DAGs) of tasks.
The easy-to-use Data Detective Airflow scheduler makes it possible to run tasks and save results into works.
Work storage support for s3, ftp, local disk and database is also included.More information about Data Detective AirflowInstallationInstall fromPyPipipinstalldata-detective-airflowSee example DAG-s indags/dagsfolder.
Package: datadict
DataDictTreat dataclasses like dictionariesInstalling DataDictDataDict is available atPyPI. You can install it using Pip:$ python -m pip install datadictUsing DataDictYou can use a DataDictdataclassas a drop-in replacement fordataclasses.dataclassin the standard library:fromdatadictimportdataclass@dataclass(order=True)classPoint:x:inty:intFor instances of the new data class, you can access attributes using square brackets, similar to dictionary access:>>>point=Point(1,2)>>>pointPoint(x=1,y=2)>>>point.x1>>>point["x"]1>>>point["y"]=3>>>pointPoint(x=1,y=3)You can also convert the dataclass instance to a proper dictionary:>>>point.asdict(){'x':1,'y':3}Installing From SourceYou can always download thelatest version of DataDict from GitHub. DataDict usesFlitas a build tool.To install DataDict from the downloaded source, run Flit throughpip:$ python -m pip install .If you want to change and play with the DataDict source code, you should install it in editable mode:$ python -m pip install -w .[dev,test]
Package: datadiff
DataDiff is a library to provide human-readable diffs of python data structures.
It can handle sequence types (lists, tuples, etc), sets, and dictionaries.Dictionaries and sequences will be diffed recursively, when applicable.It has special-case handling for multi-line strings, showing them as a typical unified diff.Drop-in replacements for some nose assertions are available.  If the assertion fails,
a nice data diff is shown, letting you easily pinpoint the root difference.datadiffworks on Python 2.6 through Python 3.DataDiff project homepage:http://sourceforge.net/projects/datadiff/ExampleHere’s an example:>>> from datadiff import diff
>>> a = dict(foo=1, bar=2, baz=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
>>> b = dict(foo=1, bar=4, baz=[0, 1, 2, 3, 4, 5, 6, 7, 8])
>>> print diff(a, b)
--- a
+++ b
{
-'bar': 2,
+'bar': 4,
 'baz': [
 @@ -5,11 +5,8 @@
  6,
  7,
  8,
 -9,
 -10,
 -11,
 ],
 'foo': 1,
}
>>>
>>> from datadiff.tools import assert_equal
>>> assert_equal([1, 2, 3], [1, 2, 5])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "datadiff/tools.py", line 29, in assert_equal
    raise AssertionError(msg)
AssertionError:
--- a
+++ b
[
@@ -0,2 +0,2 @@
 1,
 2,
-3,
+5,
]LicenseCopyright Dave BrondsemaLicensed under the Apache License, Version 2.0 (the “License”);
you may not use this file except in compliance with the License.
You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an “AS IS” BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
Package: datadigitizer
DescriptionDataDigitizer is a simple graphical tool for digitizing data from images.In terminal enter the following command:python-mdatadigitizerThe cursor is used to point a specific position in the graph
whereas all operations are done through keyboard combinations or through the main menu.Legend:Red crosses are data pointsBlue crosses are Xmin and XmaxGreen crosses are Ymin and YmaxCommands:<Ctrl-o> for loading image.<Ctrl-a> add data point.<Hold a+Left Click> add data point.<Left Click> select a data point.<Hold Ctrl+Left Click> multiple data point selection.<Ctrl-g> set Xmin from last data point or from selected data point.<Ctrl-h> set Xmax from last data point or from selected data point.<Ctrl-j> set Ymin from last data point or from selected data point.<Ctrl-k> set Ymax from last data point or from selected data point<Ctrl-l> set all limits from last 4 data points or from last 4 selected data points.<Ctrl-n> remove all limits.<Ctrl-z> remove last data point.<Ctrl-d> remove selected data point.<Ctrl-D> remove all data points.<Ctrl-m> compute the data points.<Ctrl-t> view data table.<Ctrl-s> save data points.<Ctrl-w> clear all.The online documentation is availablehere.InstallationSeeINSTALL.txt.DependenciesSeerequirements.txt.License informationSeeLICENSE.txtfor information on the history of this
software, terms & conditions for usage, and a DISCLAIMER OF ALL
WARRANTIES.
Package: datadiligence
datadiligenceRespect generative AI opt-outs in your ML training pipeline.datadiligenceaims to make it simple for ML practitioners to respect opt-outs in their training by
providing a consistent interface to check if a given work is opted-out using any known method. The goal of this
project is to make respecting opt-outs as painless as possible, while being flexible enough to support new opt-out
methods as they are developed.Why is this needed?ML training datasets are often harvested without consent from the data or content owners, meaning any ML models
trained with these datasets could be violating the wishes of content creators on how their content is used. With the
absence of an opt-out standard, many platforms and individuals have come up with their own methods of stating
their consent.Additionally, consent can change over time, and static datasets obviously cannot. A work which was
consenting at the time of the dataset’s creation may not be consenting at the time of training. Keeping up
with the current state of opt-outs is unrealistic for most practitioners, and so this project aims to make it
as easy as possible to respect opt-outs in your training pipeline.Basic UsageTo install:pip install datadiligenceAdd bulk pre-processing for URLs in your pipeline (requires Spawning API Key):>>> import datadiligence as dd
>>> urls = ["https://www.example.com/art-123456789.jpg", "https://www.example.com/art-987654321.jpg"]
>>> dd.filter_allowed(urls=urls)
 ["https://www.example.com/art-123456789.jpg"]
>>> dd.is_allowed(urls=urls)
 [True, False]Check HTTP responses in post-processing:>>> response = requests.get("https://www.example.com/art-123456789.jpg")
>>> is_allowed = dd.is_allowed(response=response)
True
>>> if is_allowed:
>>>     download_image(response.content)Full documentation is available onreadthedocs.Respected Opt-Out MethodsThis project currently supports the following opt-out methods:The Spawning API. Seehttps://spawning.ai/apifor more information.The DeviantArt X-Robots-Tag HTTP Headers. Seehttps://www.deviantart.com/team/journal/UPDATE-All-Deviations-Are-Opted-Out-of-AI-Datasets-934500371for more information.With these opt-out methods coming soon:C2PA/CAI metadata. Seehttps://c2pa.org/for more information.ContributingSee contribution guidelineshere.
Package: data-diode
Serial DiodeGeneric wire protocol and implementation for one-way data transfer over UART serial.UsageGeneralThe serial device can be specified with--deviceand baudrate with--baudrate.
If device is unspecified, it will attempt to guess the correct device based on platform and name.
If baudrate is unspecified, it defaults to 115200 Hz.See--helpfor detailed usage information.SendRundiode send <path/to/file>to send<file>.ReceiveRundiode listen, which will listen forever. When it receives a valid payload, it will write it toout/<file>where<file>was specified in the payload metadata.Note that on some systems this may need to be run as root to allow access to the serial port.
It is generally preferable to add the user to the appropriate group with something similar tousermod -a -G uucp <username>.Wire format┌─────┬──────────┬─────────────┬──────┬─────┬───────┐
│ SOT │ metadata │ data_length │ data │ EXT │ crc32 │
└─────┴──────────┴─────────────┴──────┴─────┴───────┘Here:metadatais 32 bytes,data_lengthis 8 bytes,dataisdata_lengthbytes,crc32is 8 bytes,SOTandEXTare the bytesx02,x03,crc32is the value of the CRC32 function acting onmetadata || data.InstallationInstall in developer mode withmake install_dev && source venv/bin/activate.Install as user withmake install.Installfrom PyPIwithpip install data-diode.Install Bash completion withsource bash_completion/diode.TestingRunmake test.For ad-hoc testing,socat -d -d pty,raw,echo=0 pty,raw,echo=0can be useful to connect two virtual serial devices.
If a physical serial device exists, connect withscreen /dev/<device_name> [<baud_rate>]where<baud_rate>is likely 115200.
Package: datadis
DatadisPython client forhttps://datadis.esInstallationFromPyPipipinstalldatadisUsage/Examplesimportasynciofromdatadisimportget_token,get_suppliestoken=asyncio.run(get_token('username','password'))supplies=asyncio.run(get_supplies(token))#[#    {#        "address": "home",#        "cups": "1234ABC",#        "postalCode": "1024",#        "province": "madrid",#        "municipality": "madrid",#        "distributor": "Energy",#        "validDateFrom": "2020/09",#        "validDateTo": "2021/09",#        "pointType": 0,#        "distributorCode": "2"#    }#]
Package: datadis-aseme
Herramienta para facilitar la interactividad con el WebService del sistema DATADISPublicación de datos de distribuidora al sistema DATADISGestión de peticionesInformaciones de:ContratoMáximas potenciaAUtoconsumoFácil de utilizarfromdatadis.webserviceimportDatadisWebServiceControllercontroller=DatadisWebserviceController()controller.autenticar(usuario,password)controller.contrato.__doc__>>>PublicarcontratoalsistemaDATADIS.Sielcontratoyaexisteenelsistema,semodificaraEnviarundiccionarioconlasclavesrequeridasindicadasacontinuaciondata:{"nif":"","nombre":"","comercializadora":"","tensionConexion":"","tarifaAcceso":"","discriminacionHoraria":"","tipoPunto":"","modoControlPotencia":"","fechaInicioContrato":"","cups":"","distribuidora":"","codigoPostal":"","provincia":"","municipio":""}method:'POST'pordefecto,utilizar'DELETE'paraeliminarcontrato>>>respuesta=controller.contrato(data)
Package: datadispatch
datadispatchdatadispatch provides a dispatch mechanism (a.k.a. polymorphism) likefunctools.singledispatch, but it operates on data instead of types. This is useful if you are creating simple data-centric applications where you want polymorphism à la carte without resorting to types or classes.datadispatch is available through pip:pip install datadispatchUsage examplefromdatadispatchimportdatadispatch@datadispatch(lambdaargs,_:args[0].get('type'))defhandle(message):raiseValueError('cannot handle message:{}'.format(message))@handle.register('ping')def_(message):return'you sent ping'@handle.register('pong')def_(message):return'you sent pong'print(handle({'type':'ping','payload':'hello'}))