Saravanan666/pypi-org huggingface dataset:
Package: datamachine
Modular NotebooksImagine an easy way to reuse Python notebooks in the cloud. datamachine is a useful Python package that enables you to import notebooks as modules. With datamachine, you can load notebooks from various sources such as local files, Colab links, and Github links, and then import them with ease. You can also organize notebooks into libraries that use simple codes for easy access to commonly used notebooks.FeaturesImport notebooks as modules: Importing notebooks as modules is a breeze with datamachine. You can import notebooks from various sources such as local files, Colab links, Github links, and HTTP links.Libraries: datamachine supports user-defined libraries that will make your reusable notebook modules much easier to publish and use by others.Notebook sources: You can currently source public notebooks from GitHub, Colab, and notebooks in your local file system.  I plan on adding secure notebook access and extending storage options to S3, Azure, GCP, and other cloud-based systems.  Please add an issue to GitHub if you're intersted in support for a particular source.InstallationTo install datamachine, run the following command:pipinstalldatamachineImporting datamachineOnce you've installed datamachine, you can import it. The convention is to name the module instancedm.importdatamachineasdmNotebook Locationsdatamachine currrently supports notebook links from Github, Google Colab, local file paths, and publichttpslinks with raw content.Importing a Notebook as a ModuleTo import a notebook as a module, use theimport_notebookfunction:nbo=dm.import_notebook("https://colab.research.google.com/drive/1y7x3BDkmaz6k93QjENanKHudLV8xB96Q?usp=sharing",)This function imports the notebook located at the specified location and returns a module reference like the native import command.  The Colab notebook used above provides sample analytics that can be accessed by using methods in the module.  For instance, you could run the following functionnbo.monthly_rulings()Conclusiondatamachine is a powerful tool for anyone who wants to reuse Python notebooks. With its flexible features and easy-to-use commands, datamachine makes it easy to import notebooks from various sources, store collections of code and their corresponding links in a library, and organize your code with an index. So why wait? Install datamachine today and take your notebook experience to the next level!
Package: data-magic
Data MagicA Python library to help you import and export data from various formats easilyInstallationInstall the package using pip:pipinstalldata-magicFeaturesFetches JSON data from a provided URL.Supports handling authentication tokens for secured endpoints.Converts JSON data to CSV or JSON file formats.Connects to an SQL database and fetches data based on provided connection parameters.Converts SQL query results to CSV or JSON file formats.Usageconvert json datafromdatamagicimportJsonCsvConverter# Create a new JsonCsvConverter instanceconverter=JsonCsvConverter(url='https://jsonplaceholder.typicode.com/todos/1',auth_token='your_auth_token')# Convert the JSON data to a CSV file and save itconverter.convert_to_file('output.csv',output_format='csv')# Convert the JSON data to a JSON file and save itconverter.convert_to_file('output.json',output_format='json')convert sql datafromdatabase_converterimportDatabaseToCSVConverter# Replace these with your actual database connection parametersconnection_params={'host':'your_host','user':'your_username','password':'your_password','database':'your_database'}converter=DatabaseToCSVConverter(connection_params)# Define your SQL query to retrieve data from the databasesql_query="SELECT * FROM your_table"# Convert fetched data to CSV formatconverter.convert_to_file(sql_query,'output.csv',output_format='csv')# Convert fetched data to JSON formatconverter.convert_to_file(sql_query,'output.json',output_format='json')
Package: datamaker-backup
Installationpipinstalldatamaker-backup.env.env.dist에 아래 내용 추가# ENVIRONMENT
...
DEPLOYMENT_TARGET=development|test|production

# BACKUP
ENABLE_RESTORE=true|false
ENABLE_BACKUP=true|false
RESTORE_DATA_ONLY=true|false
RESTORE_OPTIONS=do-not-set-unless-you-know-what-you-are-doing
DBBACKUP_CLEANUP_KEEP=10settings.pysettings.py에 아래 내용 추가fromdatamaker_backupimportget_backup_settings...INSTALLED_APPS=(...'dbbackup',# django-dbbackup)...# datamaker-backup# https://github.com/datamaker-kr/datamaker-backupBACKUP_CONFIG=get_backup_settings('<repository-name>',env)vars().update(BACKUP_CONFIG)postgresqlPostgreSQL을 사용하는 프로젝트는pg_dump,pg_restore로백업 및 복원이 이루어지기 때문에 postgresql client 설치를 진행해야 합니다.prepare to install latest postgresqlsudosh-c'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'wget-qO-https://www.postgresql.org/media/keys/ACCC4CF8.asc|sudotee/etc/apt/trusted.gpg.d/pgdg.asc&>/dev/nullsudoaptupdateinstall postgresql clientsudoaptinstallpostgresql-clientUsage백업databasepythonmanage.pydbbackupmediapythonmanage.pymediabackup복원databasepythonmanage.pydbrestoremediapythonmanage.pymediarestore특정 백업으로 부터 복원하는 경우pythonmanage.pydbrestore-i<backup_id>pythonmanage.pymediarestore-i<backup_id>Configuration.env에서 아래 설정을 필요에 따라 적용.DEPLOYMENT_TARGET배포 환경 이름. 주로development|test|production로 설정.Required:trueENABLE_RESTOREtrue|falsedbrestore,mediarestore사용 가능 여부. 주로 production 환경에서는 비활성화.Default:falseENABLE_BACKUPtrue|falsedbbackup,mediabackup사용 가능 여부. 주로 development 환경에서는 비활성화.Default:falseRESTORE_DATA_ONLYtrue|falsedbrestore시 schema는 그대로 두고 data만 복원할지 여부.아직까지는 해당 옵션을 활성화 할 경우 복원할 db에 data가 모두 삭제된 상태로 두어야 함.Default:falseRESTORE_OPTIONScomma separated list of pg_restore option namesExample:clean,createDefault:RESTORE_DATA_ONLY가true일 때에는data-only,disable-triggers,exit-on-errorRESTORE_DATA_ONLY가false일 때에는clean,create,if-exists,exit-on-errorDBBACKUP_CLEANUP_KEEPdbbackup시 지정된 숫자를 초과한 나머지 백업은 삭제됨.Default:30
Package: datamallet
datamalletDatamallet is a collection of helpful functions and modules built by Data scientists for Data scientists, to help
expedite the data science workflow.From a technical standpoint, datamallet is built on top of the following libraries:Scikit-learn (for creating the transformer classes).Plotly (for the automatic visualization function).Pandas (for creating scikit-learn compatible transformers,
and for creating utility functions for wrangling data).NumpyScipy.The goal of this project is to help Data scientists become more efficient in their roles by
providing commonly used functionality that have been battle tested and have been contributed by
other Data scientists.Installationdatamallet is available on pip and can be installed using the command below:pip install datamalletTestsfrom the main directory, you can run the tests by simply running the pytest command.pytestQuick startAutomatic Data Visualizationfrom datamallet.visualization import AutoPlotimport plotly.express as pxfrom datamallet.visualization import AutoPlottips = px.data.tips()autoplot = AutoPlot(df=tips,include_scatter=True,include_pie=True,include_box=True, include_sunburst=True, include_violin=True, include_treemap=True, include_histogram=True, include_correlation=True, create_html=True, filename='autoplot')list_of_charts = autoplot.show()An html file with filenameautoplot.htmlwould be created (check the current directory, a sample is also
found in this repo),
the show method also creates a list of the plotly graph objects, so you have the option of not creating the chart but
using the list of graph objects to display the charts.for chart in list_of_charts:chart.show()Modulesdatamallet currently has the following modulesVisualizationmodule which contains helper functions
for automatic visualization and for creating different types of charts such as:-Scatter plots.-Correlation plots.-Histogram.-Box plots.-Violin plots.-Treemaps.-Sun burst Charts.-Pie Charts.-Density Contour Charts.-Density Heatmap.All these charts can be created automatrically using theAutoplotclass in the visualization module,
they can also be created using individual functions in theplotmodule.Tabularmodule contains scikit-learn compatible transformers for data manipulation for tabular data,
(data which can be found in a table (pandas dataframe) either pure tabular or timeseries).
The classes found in the tabular module can be used in a scikit-learn pipeline.TheTabularmodule contains the following submodules:-featureswhich contains scikit-learn compatible transformer classes for creating new features
(more classes are welcome).-timeserieswhich contains transformers for manipulating time series data.-utilswhich contains helper functions for data wrangling and carrying out checks.preprocesswhich contains transformers for preprocessing data.
Package: data-management-demo
data-management-demoBoilerplate for data management demo python packageFree software: MIT licenseDocumentation:https://data-management-demo.readthedocs.io.FeaturesTODOCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.0.1 (2020-09-07)First release on PyPI.
Package: dataManager
DataManagerA simple library to simplify data handling in deep learning environments. With an API non dissimilar to PyTorch DataLoaderUsagefromdataManagerimportManagerasDataManagerdataFunction=lambdax:(reduceImageSize(inputImages,x),inputLabels[x],...)# Initialize Data ProvidersdataManger=DataManger(data=dataFunction,bz=32,stochasticSampling=True,indexingShape=[dataInput.shape[0]])# provide the data for the tenth batchi=10dataManger(i,stochastic=False).shape# > (32, ...)API__init__(data, indexingShape = None, bz = 32, stochasticSampling = True)data: numpy.array or function: Data to be used during training. If a numpy array is provided the data will be wrapped in a function. The input function should only take one argument which are the indexes of the elements for the data batch.indexingShape: array: Array of size 1 or 2. Used to provide the data at each batch. If the array had dim 1, then a list of indexes will be provided. IfstochasticSamplingis set to false then the indexes will be ordered from 0 to  (indexingShape - indexingShape%bz), else if the array is dim 2 the indexes will be provided from left to right and from top to bottom of the 2d matrix, in this case we also only provide as many indexes as is possible to provide with full batches.stochasticSampling: bool: if true, the indexes provided are sampled in a totally random way with equal probability for each element. For each batch all indexes will be different, but uniqueness of indexes is not guaranteed though batches.__call__(i, stochastic = None)i: integer: index of the current batch, starting from 0stochastic: bool: if stochasticSampling is to be used or notNotes:StochasticSampling overwrites ordered indexing.Undocumented Functions:StochasticSampling overwrites ordered indexing.self.getBatch(self, step, bz): return an ordered batch of indexes at a given stepself.getBatch(self, step, bz): return an ordered batch of indexes at a given stepself.getStochasticBatch(self, shape, bz): return a randomly picked set of elements with equal probability. Shape is the specific shape defaults to indexingShape if nothing given.
Package: datamanipy
datamanipyWhat is it?datamanipyis a Python package that provides tools to help you manipulating data.Where to get it?The source code is currently hosted onGitHub.
Distribution files are available on PyPI.# install datamanipy from PyPIpipinstalldatamanipyDependenciessqlalchemypandaskeyringpyreadstatopenpyxldatetimeContributing to artAll contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
Package: datamapplot
DataMapPlotCreating beautiful plots of data maps. DataMapPlot is a small library designed to help you make beautiful data map
plots for inclusion in presentations, posters and papers. The focus is on producing static plots,
or simple interactive plots, that are great looking with as little work for you as possible. All you need to do is
label clusters of points in the data map and DataMapPlot will take care of the rest. While this involves automating
most of the aesthetic choices, the library provides a wide variety of ways to customize the resulting
plot to your needs.Static Plot ExamplesSome examples of the kind of output that DataMapPlot can provide.A basic plot, with some highlighted labels:Using darkmode and some custom font choices:Alternative custom styling:Custom arrow styles, fonts, and colour maps:Interactive Plot ExamplesSome example videos of interacting with the interactive html plots.Basic UsageDataMapPlot is very easy to use. There are essentially only two functions:create_plotandcreate_interactive_plot. They take coordinates of a data map, and an array or list of labels
for the data points. A variety of further options can be used to customise the output.
A basic example might look something like:importdatamapplotdatamapplot.create_plot(data_map_coords,data_map_labels,**style_keywords)Please see the documentation for full details on usage and options.DocumentationFull documentation for DataMapPlot is available onReadTheDocs.InstallationDataMapPlot requires a few libraries, but all are widely available and easy to install:NumpyMatplotlibScikit-learnPandasDatashaderScikit-imageNumbaRequestsJinja2To install DataMapPlot you can use pip:pipinstalldatamapplotor use conda with conda-forgecondainstall-cconda-forgedatamapplotLicenseDataMapPlot is MIT licensed. See the LICENSE file for details.Help and SupportDocumentation is atRead the Docs.
The documentationincludes a FAQthat
may answer your questions. If you still have questions then pleaseopen an issueand I will try to provide any help and guidance that I can. Please read
thecode of conductfor acceptable behaviour in issue and PR discussions.ContributingContributions are more than welcome! If you have ideas for features or projects please get in touch. Everything from
code to notebooks to examples and documentation are allequally valuableso please don’t feel you can’t contribute.
To contribute pleasefork the projectmake your
changes and submit a pull request. We will do our best to work through any issues with you and get your code merged in.We would like to note that the DataMapPlot package makes heavy use of
NumFOCUS sponsored projects, and would not be possible without
their support of those projects, so pleaseconsider contributing to NumFOCUS.
Package: datamaps
![Tests](https://github.com/hammerheadlemon/datamaps/workflows/Tests/badge.svg)## Introductiondatamapsis a command line tool, written in Python, which facilitates the collection of data using Excel as the ‘form’. It accepts that Excel files are freely distributable, easy to understand and already well integrated in the non-technical office environment.The tool uses a concept adatamap, which is simple, user-created CSV file which maps cell references within Excel ‘templates’. The datamap concept makes it easy to design pretty Excel spreadsheets with blank cells (the ‘template’) intended for the user to complete - the cell references in the datamap inform thedatamapsprogram where it should expect to find user-entered data. At this point, the output is anothermasterspreadsheet which contains all the data from each completed template compiled into a single table. This is then used by other tools for further analysis.The documentation found at [Datamaps - Twenty Four Software](https://datamaps.twentyfoursoftware.com) gives a more detailed explanation.## Installationpip install datamaps## Bugs and issuesPlease report bugs as [Github issues](https://github.com/yulqen/datamaps/issues).
Package: datamarket
DataMarket scraping coreUtilities that integrate advance scraping knowledge into just one library.InstallationTo install this library in your Python environment:pip install datamarketDocumentationThis library has built functionalities for the following topics:Databases: through sqlalchemy it allows to insert records and perform queries in any database.Proxies: wide range of functions to perform HTTP requests through custom proxies or the Tor network.Tinybird: a Python client for this popular API.Drive: functions to upload, delete or authenticate to Google Drive.FTP: functions to upload, delete or authenticate to an FTP, SFTP or FTPS server.Selenium: wrapper for the main Selenium functions.
Package: datamart-augmentation
This library performs data augmentation between datasets from Auctus, NYU’s dataset search engine. You can use it to augment a dataset with a search result directly on your side without relying on the server. It is also used internally by the service to perform augmentations (the/augmentendpoint downloads the dataset using this library, performs augmentation, then sends the result to you).See also:The datamart-rest library for search/augmentationThe datamart-profiler library, used to profile datasets for searchThe datamart-materialize library, used to materialize dataset from search resultsAuctus, NYU’s dataset search engineOur project on GitLab
Package: datamart-geo
This package contains geospatial data and associated facilities to resolve administrative areas.To use the fuzzy-search capabilities (GeoData.resolve_name_fuzzy()), you will need to installdatamart-geo[fuzzy].Example usage:>>> geo_data = datamart_geo.GeoData.download()  # Download data if needed

>>> france = geo_data.resolve_name('France')
>>> france
<datamart_geo.Area "Republic of France" (3017382) type=Type.ADMIN_0>
>>> france.latitude, france.longitude
(46.0, 2.0)
>>> france.bounds
(-61.797841, 55.854503, -21.370782, 51.087541)
>>> assert france.type == datamart_geo.Type.ADMIN_0
>>> assert france.type == datamart_geo.Type.COUNTRY

>>> cuers = geo_data.resolve_name('Cuers')
>>> cuers
<datamart_geo.Area "Cuers" (6451482) type=Type.ADMIN_4>
>>> cuers.latitude, cuers.longitude
(43.2375, 6.07083)
>>> cuers.get_parent_area()
<datamart_geo.Area "Arrondissement de Toulon" (2972326) type=Type.ADMIN_3>

>>> [
...     # Show multiple results with their parent ADMIN_1
...     (a, a.get_parent_area(datamart_geo.Type.ADMIN_1))
...     for a in geo_data.resolve_name_all('Var')
... ]
[(<datamart_geo.Area "Var" (2970749) type=Type.ADMIN_2>,
  <datamart_geo.Area "Provence-Alpes-Côte d'Azur" (2985244) type=Type.ADMIN_1>),
 (<datamart_geo.Area "Vars" (6427887) type=Type.ADMIN_4>,
  <datamart_geo.Area "Nouvelle-Aquitaine" (11071620) type=Type.ADMIN_1>),
 (<datamart_geo.Area "Vars" (6442138) type=Type.ADMIN_4>,
  <datamart_geo.Area "Bourgogne-Franche-Comté" (11071619) type=Type.ADMIN_1>)]This package was built for Auctus, the dataset search engine from NYU, to resolve named areas during profiling.See also:The datamart-profiler library, used to profile datasets for searchAuctus, the dataset search engine from NYUOur project on GitLabData licenseIf you are downloading data from the default location (usingdownload()without asourceparameter), that data is distributed under specific license terms:Copyright 2021, New York University
This dataset is open data, made available under the Open Database License: http://opendatacommons.org/licenses/odbl/1.0/.
Includes data from GeoNames and sources, listed at https://www.geonames.org/datasources/
Includes map data from OpenStreetMap (r), copyright OpenStreetMap contributors and sources, listed at https://www.openstreetmap.org/copyright
Package: datamart-materialize
This library can materialize datasets from Auctus, NYU’s dataset search engine. You can use it to materialize search results directly on your side without relying on the server. It is also used internally by the service to materialize datasets (the/downloadendpoint downloads the dataset using this library then sends it to you).See also:The datamart-rest library for search/augmentationThe datamart-profiler library, used to profile datasets for searchThe datamart-augmentation library, used to performs data augmentation with a dataset from AuctusAuctus, NYU’s dataset search engineOur project on GitLab
Package: datamart-nyu
This is a specialization of the Datamart REST API wrapper for NYU’s system. It provides some optimization, running some computations locally rather than relying on the API server for everything.See also thegeneric datamart-rest wrapper, and theREST API documentation.How to install?$ pip install datamart-nyuHow to use?We provide an example using theNY Taxi Demand data. The example is availablehere.
Package: datamart-profiler
This library can profile datasets for use with Auctus, NYU’s dataset search engine. You can use it to profile datasets on your side and send that to the server for search, instead of uploading the whole dataset. It is also used internally by the service to process search-by-example queries (when sending a file to the/searchendpoint) and to add datasets to the index (to be queried against later).See also:The datamart-rest library for search/augmentationThe datamart-materialize library, used to materialize dataset from search resultsThe datamart-augmentation library, used to performs data augmentation with a dataset from AuctusAuctus, NYU’s dataset search engineOur project on GitLab
Package: datamart-rest
Datamart provides both a Python API and a REST API. This is a wrapper which exposes the REST API as the Python one, for use by clients which don’t have Datamart locally available, or with Datamart systems that use REST natively (such as NYU’s).See also theREST API documentation.How to install?$ pip install datamart-restHow to use?We provide an example using theNY Taxi Demand data. The example is availablehere.
Package: datamask
pgdatacleanerPurposeMask sensetive data in a database (i.e. PII/PHI) so it can be used for development/testing purposes.
It is meant to keep IDs, so databases that for some reason has PII data in PK/FKs won't work very well.UsageFirst, create a data dictionary from an existing database:datadict'postgresql://<user>:<password>@<host>/<database>'<schema>my_pii_dd.csvYou will need to edit this file and setpiitoyesfor any columns that need to be masked,
and thepii_typeto one of the available types to generate fake data.dataclean -hwill
list all available fakers.Then, rundatacleanto modify a database masking data as specified in the CSV. This is not a very fast operation
on databases of any significant size.dataclean-f'postgresql://<user>:<password>@<host>/<database>'-fmy_edited_pii_dd.csvIf you change your schema, adding/modifying/deleting columns and tables, you can regenerate the data dictionary
using your last copy as a seed so you don't have to re-specify columns that have not changed:datadict'postgresql://<user>:<password>@<host>/<database>'<schema>-imy_existing_dd.csvmy_new_pii_dd.csvStatusStable, supports postgresql and sqlite3.
Consists of 45% todo's and hacks - still works.Release history* 1.1.4
  * Added static_str
  * Handle lists and dicsts as JSONB [] when writing back rowsCaveatsI'm not responsible for your data. Never run this against a production database, unless you feel like testing your backup restore procedures.LicenseCopyright (c) 2021, Fredrik HåårdDo whatever you want, don't blame me. You may also use this software as licensed under the MIT or BSD licenses, or the more permissive license below:Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.