Saravanan666/pypi-org huggingface dataset:
Package: datacoco-batch
Batch is a simple interface for managing the state of jobs and workflows in batchy microservice.Installationdatacoco-batch requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-batchQuickstartself.batchy = Batch(
            wf=test_workflow,
            server="server.com",
            port="80",
        )

self.batchy.open()

self.batchy.get_status()

self.batchy.close()Sample output{
    "global": {
        "batch_id": "123456789",
        "status": "success",
        "failure_cnt": 0,
        "open_cnt": 2,
        "batch_start": "2020-01-17T06:58:01.234567",
        "batch_end": "2020-01-17T07:18:08.012345"
        }
    }DevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements-dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco_batch are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-cloud
Datacoco-cloud contains interaction classes S3, Athena, SES, SNS, SQS, ECS, EMR, Cloudwatch logsInstallationdatacoco-cloud requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-cloudUsageS3toS3InteractionPlease take note that all AWS Permissions inIAMandBucket Policiesneed to be properly in place for this utility to work.Click here for more details.Sample Code# Import the class first
from datacococloud.s3_to_s3_interaction import S3toS3Interaction

# Instantiate with your key pairs
s3toS3 = S3toS3Interaction(<source_aws_key>,
                           <source_aws_secret>,
                           <target_aws_key>,
                           <target_aws_secret>,
                           <source_aws_region>(optional default='us-east-1'),
                           <target_aws_region>(optional default='us-east-1')
                           )

# Copying the files
s3toS3.duplicate_objects(<source_bucket>,
                         <target_bucket>,
                         <source_bucket_prefix>,
                         <target_path>,
                         <source_bucket_suffix>(optional default=''))


# Moving the files
# This deletes the file from the source after copying to the target
s3toS3.move_objects(<source_bucket>,
                         <target_bucket>,
                         <source_bucket_prefix>,
                         <target_path>,
                         <source_bucket_suffix>(optional default=''))Termssource_aws_key- AWS key ID from source accountsource_aws_secret- AWS key secret from source accounttarget_aws_key- AWS key ID from target accounttarget_aws_secret- AWS key secret from target accountsource_aws_region- AWS region of the sourceS3buckettarget_aws_region- AWS region of the sourceS3bucketsource_bucket- S3 bucket of the source filetarget_bucket- S3 bucket where the files are going to be transferredsource_bucket_prefix- The prefix of the files to transfer from the sourceNote:Add/at the end to specify a folder e.g (files/)target_path- the Path at the target bucket where the files will be transferredNote:if the the folder does not exist, it will auto create it for yousource_bucket_prefix- The suffix of the files to transfer from the sourceQuickstartpython3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install --upgrade pip
pip install -r requirements_dev.txtDevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements_dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco_cloud are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-core
Equinox Common Code Utility for Python 3 with minimal dependencies and
easy installation!Includes utilities for logging and config filesInstallationTo install the latest version, use pip:pip install datacoco-coreLoggerThe logging module is a lightwight wrapper around the default logging
module. Basic usage:from datacoco_core import Logger
l = Logger()
l.l("Your important log message here')By default the log message will be saved in a logs dir of the project
root, in a file named by python module and date. For example:cat logs/test.py-20190827-135736.log
2019-08-27 13:57:36,471 Your important message hereOptionally, parameterslognameandproject_namecan be passed on
class instantiation to customize the logfile name, and logfile path
respectively.configThe config module is a lightweight wrapper around the configparser
module. It converts INI files to a dictionary object. By default, the
Config class will look for a file named inetl.cfgin the project
root.Example INI config file:[secret1]
answer_to_the_universe=42This credential can be accessed by the following example code:from datacoco_core.config import config
c = config()
c['secret1']['answer_to_the_universe']
42Note: The config class assumes base64 hashing for any key named pwd or
password. (See below)Password EncryptionThe coco config class uses base64 encryption. Any option named pwd or
password will be assumed base 64 encrypted. To derive the encrypted
password for your config, launch python shell and run the following
command:>>> import base64
>>> print base64.b64encode("password")
cGFzc3dvcmQ=In python3 if you get the errorTypeError: abytes-likeobject is required, not 'str'do this.>>> import base64
>>> print(base64.b64encode(b'password'))
b'cGFzc3dvcmQ=DevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment
for development:python3 -m venv <virtual env name>
source venv/bin/activate
pip install --upgrade pip
pip install datacoco-coreTestingTo run the testing suite, simply run the command:tox
Package: datacoco-db
Equinox Common Code Utility for Python 3 for DB interactions! There are
currently interaction classes for the following DBs and Apps:MSSQLMySQLSQLitePostgresRedshiftQuick StartSample Usagefrom datacoco_db import MSSQLInteraction

mssql = MSSQLInteraction(dbname="db_name",
                        host="server",
                        user="user",
                        password="password",
                        port=1433)

mssql.conn() # open a connection

mssql.batch_open() # cursor

results = mssql.fetch_sql_one("SELECT * FROM MyTable") # fetch one

print(results)

mssql.batch_close() # close cursorThe example above makes use of mssql_tools.
All tools follows the same pattern in terms of usage.Installationdatacoco-db requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-dbDevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtPyodbc Dependency InstallationInstalling the Microsoft ODBC Driver for SQL Server on Linux and macOShttps://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-ver15Testingpip install -r requirements-dev.txtModify the connection configuration for integration testing.To run the testing suite, simply run the command:python-munittest discover testsFor coverage report, runtoxView the results in
.tox/coverage/index.htmlContributingContributions to datacoco_db are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-email-tools
datacoco-email_tools provides basic interaction with the Amazon Web
Service (AWS) Simple Email Service.Installationdatacoco-email_tools requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-email-toolsQuickstartSending email using one line wrapperEmail.send_mail(
    aws_access_key=AWS_ACCESS_KEY,
    aws_secret_key=AWS_SECRET_KEY,
    aws_sender=SENDER,
    aws_region=AWS_REGION,
    to_addr=RECIPIENTS,
    subject='hello world',
    from_addr='email@github.com',
    text_msg='text message',
    html_msg='html message')Sending email with attachment using one line wrapperEmail.send_email_with_attachment(
    aws_access_key=AWS_ACCESS_KEY,
    aws_secret_key=AWS_SECRET_KEY,
    aws_sender=SENDER,
    aws_region=AWS_REGION,
    to_addr=RECIPIENTS,
    subject='hello world',
    from_addr='email@github.com',
    filename='helloworld.txt',
    body_msg='email body text')DevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements-dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco_email_tools are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-ftp-tools
datacoco-ftp_tools provides basic interaction for FTP (File-Transfer-Protocol)
Standard Internet protocol for transmitting files between computers on the Internet over TCP/IP connections
This module has FTP and SFTP supportInstallationdatacoco-ftp_tools requires Python 3.6+python3 -m venv venv
source venv/bin/activate
python -m pip install datacoco_ftp_toolsQuickstartSFTP and write file to remote server using FTPfrom datacoco_ftp_tools import FTPInteraction


# Sample Code for FTP Interaction

ftp = FTPInteraction('ftp',ftp_site,ftp_user,ftp_password)

ftp.conn()

ftp.curr_dir() # outputs '/web_analytics'

ftp.call_dir('Monitoring') # outputs '/web_analytics/Monitoring'

ftp.write_file('test.txt', 'test')

ftp.quit()SFTP and write file to remote server using SFTPfrom datacoco_ftp_tools import FTPInteraction


# Sample Code for SFTP Interaction

sftp = FTPInteraction('sftp',sftp_site,sftp_user,sftp_password)

sftp.conn()

sftp.curr_dir() outputs '/web_analytics'

sftp.call_dir('Monitoring') outputs '/web_analytics/Monitoring'

sftp.write_file('test.txt', 'test')

sftp.quit()SFTP and write file to remote server using SFTPfrom datacoco_ftp_tools import SFTPInteraction


# Sample Code for SFTP Interaction

sftp = SFTPInteraction(sftp_site, user, None, key_file='key.ppk')

sftp.conn()

sftp.call_dir('Monitoring') outputs '/web_analytics/Monitoring'

sftp.write_file(schema[table], remote_path=path)

sftp.quit()DevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements-dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco_ftp_tools are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-redis-tools
datacoco-redis_tools provides basic interaction to redis databaseInstallationdatacoco-redis_tools requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-redis_toolsQuickstartself.rconn = RedisInteraction(
    host=<HOST>,
    port=<PORT>,
    db=<DB>,
    decode_responses=True,
)

self.rconn.connect()
self.rconn.set_key('key', 'key_value')DevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements-dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco-redis_tools are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacoco-sf-tools
Installationdatacoco-sf_tools requires Python 3.6+python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install datacoco-sf-toolsQuickstartDevelopmentGetting StartedIt is recommended to use the steps below to set up a virtual environment for development:python3 -m venv <virtual env name>
source <virtual env name>/bin/activate
pip install -r requirements.txtTestingpip install -r requirements-dev.txtTo run the testing suite, simply run the command:toxorpython-munittest discover testsContributingContributions to datacoco_sf_tools are welcome!Please reference guidelines to help with setting up your development
environmenthere.
Package: datacollector
Data CollectorBy QueryTo get the papers data by search query, you can use the following python script,from datacollector import datacollector_query

# Search for a query, biofilm co2, with 10 results and download the pdf format of the result 
datacollector_query('biofilm co2', 10, True)

# Search for a query, biofilm co2, with 10 results and without download the pdf format of the result 
datacollector_query('biofilm co2', 10)In this script, it takes 3 arguments.Search Query (Non-empty String)Number of results (Integer)Download PDF file? (yes/no)NOTE: For the search query, it must be a non-empty String, and if you want more than one query, using + symbol between two words (Ex. word1+word2).
This script willSearch for the query at the PubMed repositoryFormat the result as JSON formatName the JSON with paper’s id (Ex. 1234.json)Automatucally create a directory (./datasets_query/json) to save the JSON resultsFor the pdf file, they will be saved into ./pdfThe JSON file will contain the metadata of the paperpmidauthorsyeartitleabstractdoireferencescitedbyIf you type yes for the third argument, it also will download the paper with pdf format into the pdf directory from a repository HERE.NOTE: You might not be able to download some of the paper with pdf format. The reason I have found is first the repository doesn’t have it. Second, the paper hasn't been published yet because it’s too new.By PMIDsTo get the papers data by list of ids, you can use the following python script,from datacollector import datacollector_query

# import list of ids with text format, download metadata, and download the pdf format of the result 
datacollector_query('./ids_list.txt', True)

# import list of ids with text format, download metadata without downloading the pdf format of the result 
datacollector_query('./ids_list.txt')In this script, it takes 2 arguments.File path (TXT format)Download PDF file? (yes/no)This script will do the same things as ABOVE, but with list of pmids, not query. It also will automatically create a directory (./datasets_id/json) to save the JSON results.
NOTE: For the TXT file, you must separate those pmids with a new line character (“\n”).
Package: datacollector-api-client
A thin client to send log data to Azure Monitor by using theHTTP Data Collector API.Installation$ pip install datacollector-api-clientUsagefrom datacollector_api_client.client import DataCollectorWrapper

wrapper = DataCollectorWrapper(LOGANALYTICS_WORKSPACE_ID, LOGANALYTICS_WORKSPACE_KEY)

data = [{
   "application": "my_app",
   "message": "my log message"
}]

response = wrapper.log_info(structured_log_message=data)
print(response)To log a message as error, use:wrapper.log_error(structured_log_message=data)You can also override the default name of the tables by setting parameter:log_type=”your_preferred_table_name”Enriching structured logging with Databricks informationPass your dbutils instance from your Databricks session to the librarywrapper = DataCollectorWrapper(LOGANALYTICS_WORKSPACE_ID, LOGANALYTICS_WORKSPACE_KEY, dbutils)

data = [{
   "application": "Notebook",
   "message": f'Number of rows in dataframe: {df.count()}'
}]

response = wrapper.log_info(structured_log_message=data)
print(response)The following data is also collected and appended to your log:dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get()dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply(‘clusterId’)dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply(‘sessionId’)dbutils.notebook.entry_point.getDbutils().notebook().getContext().tags().apply(‘user’)
Package: datacommons
Data Commons Python APIThis is a Python library for accessing data in the Data Commons Graph.See also:Data Commons Pandas API.To get started, install this package from pip.pip install datacommonsOnce the package is installed, importdatacommons.import datacommons as dcIf you would like to provide an API key, follow the steps inSetting up access to the Data Commons API,
add the following line to your code:dc.set_api_key('YOUR-API-KEY')Data Commonsdoes not chargeusers, but uses the API key for
understanding API usage.For more detail on getting started with the API, please visit ourAPI Overview.When you are ready to use the API, you can refer todatacommons/examplesfor
examples on how to use this package to perform various tasks. More tutorials and
documentation can be found on ourtutorials page!About Data CommonsData Commonsis an open knowledge repository that
provides a unified view across multiple public data sets and statistics. You can
view whatdatasetsare currently ingested
and browse the graph using ourbrowser.LicenseApache 2.0DevelopmentThe Python API currently supportspython>=2.7.To test, run:$ ./run_tests_local.shTo debug the continuous integration tests, run:$ cloud-build-local --config=cloudbuild.yaml --dryrun=false .Both commands will run the same set of tests.To run the examples:$ python -m datacommons.examples.XXXwhere XXX is the module you want to run.ReleaseNote: Always releasedatacommons_pandaswhendatacommonsis released.If this is your first time releasing to PyPI, please review the PyPI guide
starting from thesetup section.Release to Test PyPIInsetup_datacommons.pyandsetup_datacommons_pandas.py:Append "-USERNAME" to the package "NAME". For example,NAME = 'foo_package-janedoe123'.Increment the "VERSION" codes to something that has not been used in your
test project. This will not affect the production PyPI versioning.Build the dists:rmdist/*
python3-mpipinstall--user--upgradesetuptoolswheel
python3setup_datacommons.pysdistbdist_wheel
python3setup_datacommons_pandas.pysdistbdist_wheelRelease the dists to TestPyPI:python3-mpipinstall--user--upgradetwine
python3-mtwineupload--repositorytestpypidist/*Release to Production PyPIInsetup_datacommons.pyandsetup_datacommons_pandas.py:Revert the package name todatacommonsanddatacommons_pandasUpdate and double check "VERSION"UpdateCHANGELOG.mdanddatacommons_pandas/CHANGELOG.mdBuild the dists:rmdist/*
python3-mpipinstall--user--upgradesetuptoolswheel
python3setup_datacommons.pysdistbdist_wheel
python3setup_datacommons_pandas.pysdistbdist_wheelRelease the dists to PyPI:python3-mpipinstall--user--upgradetwine
twineuploaddist/*SupportFor general questions or issues about the API, please open an issue on ourissuespage. For all other
questions, please send an email tosupport@datacommons.org.Note- This is not an officially supported Google product.
Package: datacommons-pandas
Data Commons Pandas APIThis is a Python library for creating pandas objects with data in the
Data Commons Graph.To get started, install this package from pip.pip install datacommons_pandasOnce the package is installed, importdatacommons_pandas.import datacommons_pandas as dcpdFor more detail on getting started with the API, please visit ourAPI Overview.When you are ready to use the API, you can refer todatacommons_pandas/examplesfor
examples on how to use this package to perform various tasks. More tutorials and
documentation can be found on ourtutorials page!About Data CommonsData Commonsis an open knowledge repository that
provides a unified view across multiple public data sets and statistics. You can
view whatdatasetsare currently ingested
and browse the graph using ourbrowser.LicenseApache 2.0DevelopmentPlease follow theDevelopment instructions.ReleasePlease follow theRelease instructions.SupportFor general questions or issues about the API, please open an issue on ourissuespage. For all other
questions, please send an email tosupport@datacommons.org.Note- This is not an officially supported Google product.
Package: datacomp
DataComp: A Python Framework for Systematic Dataset ComparisonsDescriptionDataComp is an open source Python package for domain independent multimodal longitudinal dataset comparisons.
It serves as an investigative toolbox to assess differences between multiple datasets on feature level.
DataComp empowers data analysts to identify significantly different and not significantly difference between datasets and thereby is helpful to identify comparable dataset combinations.Typical application scenarios are:Identifying comparable datasets that can be used in machine learning approaches as training and independent test dataEvaluate if, how and where simulated or synthetic datasets deviate from real world dataAssess (systematic) differences across multiple datasets (for example multiple sampling sites)Conducting multiple statistical comparisonsComparative visualizationsThe figure above depicts a typical DataComp workflow.Main FeaturesDataComp supports:Evaluating and visualizing the overlap in features across datasetsParametric and nonparametric statistical hypothesis testing to compare feature value distributionsCreating comparative plots of feature value distributionsNormalizing time series data to baseline and statistically comparing the progression of features over timeComparative visualization of feature progression over timeHierarchical clustering of the entities in the data sets to evaluate if dataset membership labels are evenly
distributed across clusters or assigned to distinct clustersPerforming a MANOVA to assess the influence of features onto the dataset membershipInstallationpipinstalldatacompDocumentationThe full package documentation can be foundhere.Application examplesExample notebooks showcasing Datacomp workflows and results on simulated data can be found atDataComp_Examples:Cross-sectional Comparison ExampleLongitudinal Comparison Example
Package: datacompass
DataCompassDataCompass is a Python package designed to enhance data analysis with pandas DataFrames. It provides a set of tools to quickly inspect and understand the structure and contents of DataFrames, aiding in efficient data exploration and preprocessing.FeaturesDescribe Columns: Analyze and list numerical and categorical columns in a DataFrame.Display Missing Information: Calculate and display the count and percentage of missing values in each column.Display Basic Information: Show basic details like the number of columns, rows, and the first few observations.Display Unique Values: Enumerate unique values or the count of unique values in each column.InstallationTo install DataCompass, simply use pip:pipinstalldatacompassimportpandasaspd
fromdatacompassimportdescribe_columns,display_missing_info,display_basic_info,display_unique_values# Sample DataFramedf=pd.DataFrame({'A':[1,2,None,4],'B':['a','b','b','c']})# Describe Columnsdescribe_columns(df)# Display Missing Informationdisplay_missing_info(df)# Display Basic Informationdisplay_basic_info(df)# Display Unique Valuesdisplay_unique_values(df)### Describe Columns OutputNumberofNumericalColumns:1['A']-------------------------------------------------------------------------------------
NumberofCategoricalColumns:1['B']### Display Missing Information Output|MissingCount|MissingPercentage||----------------:|-------------------:||A|1|25.0||B|0|0.0|### Display Basic Information OutputNumberofColumns:2NumberofRows:4First6ObservationsofOurData:AB1.0a2.0b
NaNb4.0c### Display Unique Values OutputAcontains:1.0,2.0,nan,4.0
Bcontains:a,b,cRequirementspandasContributionContributions to DataCompass are welcome! Please feel free to submit a pull request or open an issue on theGitHub repository.LicenseThis project is licensed under the MIT License - see the LICENSE file for details.AuthorSai Koushik Gandikota