Saravanan666/pypi-org huggingface dataset:
Package: etl-pbshop
ETL LibThis is a simple package built for Portobello Shop integrations, and now is available as ETL lib.Can be used with several services, like:OracleSalesforceMSSQL ServerGSuiteMicrovixODBC Drivers: PYODBC and SQLALCHEMYAnd any REST services ...Comming soon:RabbitMQKafkaOpenVault (Credentials)...Basic UsageRewrite ETL ABC classUse inheritance and rewrite the three abstract methods (even if you don't use it):frometl_pbshopimportETL,ConnectorsclassMyETL(ETL):def__init__(self):self.config=MyConfiguration()self.log=self.config.logself.connector=Connectors(config=self.config,log=self.log,services={'google':'PROD',"salesforce":"PROD"})defextract(self):pass# extract the needed data using Connectorsdeftransform(self):pass# do some transformationsdefload(self):pass# upload your transformationsMain caller:On the main caller, you can simply:if__name__=='__main__':etl=MyETL()etl.config.start()try:etl.run()exit(0)exceptExceptionase:etl.get_error(f"ERROR on main:{str(e)}")raiseefinally:etl.config.finish()Rewriting default ConfigModel:In another class, you can rewrite the ConfigModel class and input your values:frometl_pbshop.config_modelimportConfigModelclassMyConfiguration(ConfigModel):def__init__(self):super().__init__(__file__)self.SQL_QUERIES={"produtos":"SELECT * FROM Products2"}self.INTEGRATION_SPREADSHEET_ID='654asd5as1das5d165a4d68'Environment variables:You can put your environment variables to use the Connectors class.
In a.envfile, set it in groups, like:INTEGRACAO_ORACLE_ID_1=1
INTEGRACAO_ORACLE_TOKEN_VALUE_1=asd1234asd==
INTEGRACAO_ORACLE_TOKEN_URL_1=http://example.com/token
INTEGRACAO_ORACLE_METHOD_URL_1=http://example.com/method_to_post
INTEGRACAO_ORACLE_USER_1=asd..
INTEGRACAO_ORACLE_PASS_1=qwerty..
INTEGRACAO_ORACLE_ENVIRONMENT_1=DEV

INTEGRACAO_MICROVIX_ID_1=1
INTEGRACAO_MICROVIX_TOKEN_VALUE_1=1234-123-123-1234
INTEGRACAO_MICROVIX_METHOD_URL_1=https://webapi.microvix.com.br/1.0/api/method
INTEGRACAO_MICROVIX_USER_1=user
INTEGRACAO_MICROVIX_PASS_1=pass
INTEGRACAO_MICROVIX_ENVIRONMENT_1=PROD

INTEGRACAO_SALESFORCE_ID_1=1
INTEGRACAO_SALESFORCE_TOKEN_VALUE_1=qwerty
INTEGRACAO_SALESFORCE_METHOD_URL_1=https://login.salesforce.com/services/Soap/u/47.0
INTEGRACAO_SALESFORCE_USER_1=username@example.com
INTEGRACAO_SALESFORCE_PASS_1=myPass
INTEGRACAO_SALESFORCE_ENVIRONMENT_1=PRODYou only need to keep the pattern:INTEGRACAO_<service_name>_<key_name>_<group_id>More info, please contact:daniel.camargo@portobelloshop.com.br
Package: etl-server
DGP UIThis library and app provide a wrapper around airflow, providing a means to add / remove DAGs (Pipelines) via a web-ui based on a configuration defining the Pipeline 'kinds' and the parameters each kind requires.Pipeline DashboardEdit/New PipelinePipeline StatusQuickstartCreate a folder containing:Aconfiguration.yamlfile with the details on your pipeline kinds, e.g.{"kinds":[{"name":"kind1","display":"Kind 1","fields":[{"name":"param1","display":"Parameter 1"},{"name":"param2","display":"Parameter 2"}]},{"name":"kind2","display":"Kind 2","fields":[{"name":"param3","display":"Parameter 3"},{"name":"param4","display":"Parameter 4"}]}],"schedules":[{"name":"monthly","display":"Monthly"},{"name":"daily","display":"Daily"}]}(If schedules are not specified, a default schedules list will be used).The Airflow DAGs Creator - a Python file that reads the pipeline configuration and creates your Airflow DAGs.
Sample code:importdatetimeimportloggingfromairflowimportDAGfromairflow.operators.bash_operatorimportBashOperatorfromairflow.utilsimportdatesfrometl_server.modelsimportModelsetl_models=Models()default_args={'owner':'Airflow','depends_on_past':False,'start_date':dates.days_ago(1),}forpipelineinetl_models.all_pipelines():# pipeline looks like this:# {#   "id": "<identifier>",#   "name": "<English Name of Pipeline>",#   "kind": "<kind-name>",#   "schedule": "<schedule>",#   "params": {#      "field1": "value1",#      .. other fields, based on kind's fields in configuration#   }# }dag_id=pipeline['id']logging.info('Initializing DAG%s',dag_id)dag=DAG(dag_id,default_args=default_args,schedule_interval=datetime.timedelta(days=1))task=BashOperator(task_id=dag_id,bash_command='echo "%s"; sleep 10 ; echo done'%pipeline['name'],dag=dag)globals()[dag_id]=dagUse adocker-composesetup to run the server, an exampledocker-compose.yamlfile:version:"3"services:db:image:postgres:12environment:POSTGRES_PASSWORD:postgresPOSTGRES_USER:postgresPOSTGRES_DB:etlsexpose:-5432volumes:-/var/lib/postgresql/dataserver:build:.image:akariv/airflow-config-uienvironment:DATABASE_URL:postgresql://postgres:postgres@db/etlsAIRFLOW__CORE__SQL_ALCHEMY_CONN:postgresql://postgres:postgres@db/etlsexpose:-5000ports:-5000:5000depends_on:-dbvolumes:-/path/to/local/dags/folder/:/app/dagsAfter running (docker-compose up -d server), open your browser athttp://localhost:5000to see the web UI.Another option is to create a new Docker image which inherits fromakariv/airflow-config-uiand replaces the contents of/app/dags/with theconfiguration.jsonfile and your DAG Python files.
Package: etl-table-converter-talenttech-oss
ETL table converterconfig-from db to db
dialect - sql-alchemy dialectsql_credentials={"pg":{"dialect":"postgresql","database":os.getenv("PG_DATABASE"),"schema":os.getenv("PG_SCHEMA"),"user":os.getenv("PG_USER"),"host":os.getenv("PG_HOST"),"port":os.getenv("PG_PORT"),"password":os.getenv("PG_PASSWORD")},"ch":{"dialect":"clickhouse+native","database":os.getenv("CH_DATABASE"),"user":os.getenv("CH_USER"),"host":os.getenv("CH_HOST"),"port":os.getenv("CH_PORT"),"password":os.getenv("CH_PASSWORD")},"vertica":{"dialect":"vertica+vertica_python","database":os.getenv("VERTICA_DATABASE"),"schema":os.getenv("VERTICA_SCHEMA"),"user":os.getenv("VERTICA_USER"),"host":os.getenv("VERTICA_HOST"),"port":os.getenv("VERTICA_PORT"),"password":os.getenv("VERTICA_PASSWORD")},}Usagepip3installetl-table-converter-talenttech-ossfromconverter.fields_converterimportFieldsConverterfrom_db="vertica"to_db="ch"converter=FieldsConverter(sql_credentials,from_db,to_db)tables=["table_name1","table_name2"]converter.create_list_of_tables(tables,dir="/")
Package: etlTest
#etlTest[![Build Status](https://travis-ci.org/OpenDataAlex/etlTest.png?branch=dev)](https://travis-ci.org/OpenDataAlex/etlTest)[![Coverage Status](https://coveralls.io/repos/OpenDataAlex/etlTest/badge.png?branch=dev)](https://coveralls.io/r/OpenDataAlex/etlTest?branch=dev)[![Codacy Badge](https://www.codacy.com/project/badge/0ff3a2e5b651416e9292ca9cdedf58f8)](https://www.codacy.com)[![Documentation Status](https://readthedocs.org/projects/etltest/badge/?version=latest)](https://readthedocs.org/projects/etltest/?badge=latest)[![endorse](https://api.coderwall.com/dbaalex/endorsecount.png)](https://coderwall.com/dbaalex)##InstallationYou can install **etlTest** by downloading the source and using the setup.py script as follows:$ git clone git@github.com:OpenDataAlex/etlTest.git$ cd etlTest$ python setup.py installThis setup call installs all of the necessary python dependencies. There are a few external dependencies as well, so please see the section below labeled "Non-Python Dependencies".Once you have done that, its ready to run!### So what is etlTest?Having come from software development and working with data integration tools, we always wondered why there wasn'tsome kind of uniform unit and integration testing tool specific to data integration.  etlTest aims to fill that gapby providing an easy to use tool and data source agnostic testing tool.  Testing is designed to be "black box" -which means that we aren't diving into the actual data integration code.  Rather,we are executing the data integration process based on test data sets provided by the test writer and comparing theresults using Python's unittest framework.etlTest is based on the work and discussions that were started with etlUnit.### QuickstartTo actually use etlTest, you need a data and test file for it to act on. A most basic resource file can befound inthe [samples](https://github.com/OpenDataAlex/etlTest/tree/dev/etltest/samples) directory of the project(data/etlUnitTest/users.yml and test/dataMart/users_dim.yml).Executing thefollowing will take that resource, generate some python code in the output directory specified, and run the code which will display the output of the tests executed to your terminal.$ python etlTest/etltest/etlTest.py -f <path to users_dim.yml test file> -o /tmp/ -g -e### DocumentationThe documentation for **etlTest** can be found on Read the Docs [here](https://etlTest.readthedocs.org/en/latest/).### Non-Python DependenciesThe only dependencies that are not handled in python currently are the ones for SQLAlchemy to connect to datasources. Documentation on how to install these is as follows:* [MySQL](https://github.com/OpenDataAlex/etlTest/blob/develop/docs/mysql_deps.md)* [Oracle](https://github.com/OpenDataAlex/etlTest/blob/develop/docs/oracle_deps.md)### Reporting IssuesWe would love some feedback! Please do not hesitate to report any issues/questions/comments via the [Github Issue Tracker](https://github.com/OpenDataAlex/etlTest/issues).
Package: etlutils
etlutilsThis is a collection of utility functions for my etl projects. Sharing as they might be useful to others as well.current versionThe current version is 0.2.2 as this is a very early release. Use at your own risk as things will change.documentationRead the documentation onreadthedocsinstalling with pippip install etlutilsbuilding for developmentmake init
pip install -e .
make test
make lintcontributionsAll contributions are welcome, please submit a PR. Issues and feature requests are fine as well, but they may take some time for me to get to. Better to submit a PR.requiresPython 3.6 or above, I'm not using any earlier versions of python on any of my projects anymore, so supporting older versions isn't high on my list of priorities.
Package: et-micc2
Micc2_is a Python project manager: it helps
you organize your Python project from simple single file modules to fully fledged
Python packages containing modules, sub-modules, apps and binary extension modules
written in Fortran or C++.Micc2organizes your project in a way that is considered good
practice by a large part of the Python community.Micc2helps you create new projects. You can start small with a simple one-file
package and add material as you go, such as:Pythonsub-modulesandsub-packages,applications, also known as command line interfaces (CLIs).binary extension moduleswritten in C++ and Fortran. Boiler plate code is
automatically added as to build these binary extension with having to go through
al the details. This is, in fact, the foremost reason that got me started on this
project: ForHigh Performance Pythonit is essential to rewrite slow and
time consuming parts of a Python script or module in a language that is made
for High Performance Computing. As figuring out how that can be done, requires
quite some effort,Micc2was made to automate this part while maintaining the
flexibility.Micc2adds typically files containing example code that shows you how to add your
own functionality.You can automaticallyextract documentationfrom the doc-strings of your files,
and build html documentation that you can consult in your browser, or a .pdf
documentation file.With a little extra effort the generated htmldocumentation is automatically publishedtoreadthedocs.Micc2helps you withversion management and control.Micc2helps you withtestingyour code.Micc2helps you withpublishingyour code to e.g.PyPI, so
that you colleagues can use your code by simply running:> pip install your_nifty_packageCreditsMicc2does not do all of this by itself. For many things it relies on other strong
open source tools. Here is a list of toolsmicc2is using or cooperating with happily:Pyenv: management of different Python versions on your desktop.Poetryfor packaging and publishing.Gitfor version control.Pytestfor testing your code.Sphinxto extract documentation from your project’s
doc-strings.CMakeis used for building binary extension modules written
in C++ and Fortran.F2pyfor transforming modern Fortran code
into performant binary extension modules interfacing nicely with numpy arrays.Pybind11as the glue between C++ source
code and performant binary extension modules, also interfacing nicely with numpy arrays.RoadmapThese features are still on our wish list:Contininous integtration (CI)Code style, e.g.flake8orblackProfilingGui for debugging C++/Fortran binary extensionsMicc2 projects on Windows (So far, only support on Linux and MacOS).
Package: etna-cli
etna-cliPython tool for my school's intranetUsageUsage: etna.py [OPTIONS] COMMAND [ARGS]...

Options:
  -v, --version  print version
  -d, --docs     open docs page
  --help         Show this message and exit.

Commands:
  config        Init, edit or delete etna config.
  conversation  Conversations on intranet.
  declare       Declaration.
  event         Events.
  gitlab        Gitlab.
  project       Projects.
  rank          Rank by promotion.
  student       Student stuff.
  task          Add quests and projects to TaskWarrior.
  ticket        Tickets.InstallationMake sure to havetaskwarriorinstalled to task related stuffGithub repository$gitclonehttps://github.com/matteyeux/etna-cli
$cdetna-cli
$poetryinstallPyPIInstallation :pip3 install etna-cliUpdate :pip3 install --upgrade etna-cliSetupMake sure to have~/.local/binin your$PATH(export PATH=$PATH:~/.local/bin)If you run etna-cli for the first time you may runetna config initto set credentials and optional Gitlab Token.$ etna config init
ETNA username : demo_t
Password:
Add Gitlab API token ? [Y/n]: Y
Gitlab API token :Password and Gitlab token are not printed to STDOUT.CreditsPowered byetnawrapper
Package: etna-dulcinea
DulcineaScript to manually load and execute jobs based on Quixote blueprints.Installpip install -r requirements.txtHow to use dulcineaIn most cases, the script can be used as follows:$ dulcinea path/to/moulinette/ --override-deliveries /path/to/deliveries/ --auto-infoFor an extensive tutorial on how Dulcinea can be used to test moulinettes, seethe relevant documentation.
Package: etna-rocinante
rocinanteInstallingRocinante depends on a working installation of Python 3.7 (or above), and Docker.Installing PythonIf you don’t have Python 3 installed on your machine yet, you can obtain it by reading theofficial installation instructions.To verify your installation, you can try the following commands in your terminal, and ensure that your versions are equivalent to or higher than those in the output below:> python3 --version
Python 3.7.3
> pip3 --version
pip 18.1 from /usr/lib/python3/dist-packages/pip (python 3.7)Notes about installing on virtual machines provided by the BluAt the moment, the configuration of the virtual machines provided by the Blu requires Python 3.7 to be built manually before installing.Thus, the following commands must be executed:sudoapt-getinstallcurlzlib1g-devlibssl-devlibffi-devlibreadline-devlibexpat-devsoftware-properties-common
curl-Ohttps://www.python.org/ftp/python/3.7.4/Python-3.7.4.tgz
tarxvfPython-3.7.4.tgzcdPython-3.7.4
./configure--prefix=/usr\--enable-shared\--with-system-expat\--with-system-ffi\--with-ensurepip=yes\--enable-optimizations
make
sudomkdir/usr/lib/python3.7/site-packages/-p
sudomkdir/usr/lib/python3.7/dist-packages/-p
sudoln-s/usr/share/pyshared/lsb_release.py/usr/lib/python3.7/site-packages/lsb_release.py
sudoln-s/usr/lib/python3/dist-packages/softwareproperties//usr/lib/python3.7/dist-packages/
sudomakeinstall
sudochmod-v755/usr/lib/libpython3.7m.so
sudochmod-v755/usr/lib/libpython3.so
curlhttps://bootstrap.pypa.io/get-pip.py-oget-pip.py
sudopython3get-pip.py--no-setuptools--no-wheel
sudopip3install--upgradepipThis might take a looooong time, go grab a coffee.Installing DockerDocker providesofficial installation guidesfor each operating system.Installing RocinanteAs soon as the dependencies are correctly installed, Rocinante can be installed usingpip:sudo pip3 install etna-rocinanteConfiguring RocinanteRocinante requires a configuration file, in order to know how to receive jobs from RabbitMQ, and how to download moulinettes from the intranet.rabbitmq:host:"172.16.128.191"port:5671username:"some_user_name"password:"some_password"virtual_host:"development"credentials:intra:username:"some_user_name"password:"some_password"gitlab:token:"aC0MpL1c4T3DT0kEn"queues:-name:"quest_moulinette"driver:"intra"result_routing_key:"quest_moulinette.result"log_directory:"/var/log/rocinante"root_directory:"/var/lib/rocinante"debug:trueadditional_docker_daemon:network_bridge_mask:"10.9.8.7/24"The path to this file must be given using the-c/--config-fileswitch.Running multiple instancesMultiple instances of Rocinante can be executed simultaneously on the same machine. For that, the--root-directoryswitch must be used to specify a custom root directory for each instance.For example, the first instance could be started with the following command:> rocinante --config-file /etc/rocinante/config.yml --root-directory /var/run/rocinante1Whereas the second instance would be launched like so:> rocinante --config-file /etc/rocinante/config.yml --root-directory /var/run/rocinante2
Package: etna-ts
ETNA Time Series LibraryHomepage|Documentation|Tutorials|Contribution Guide|Release NotesETNA is an easy-to-use time series forecasting framework.
It includes built in toolkits for time series preprocessing, feature generation,
a variety of predictive models with unified interface - from classic machine learning
to SOTA neural networks, models combination methods and smart backtesting.
ETNA is designed to make working with time series simple, productive, and fun.ETNA is the first python open source framework ofTinkoff.ruArtificial Intelligence Center.
The library started as an internal product in our company -
we use it in over 10+ projects now, so we often release updates.
Contributions are welcome - check ourContribution Guide.InstallationETNA is onPyPI, so you can usepipto install it.pipinstall--upgradepip
pipinstalletna-tsGet startedHere's some example code for a quick start.importpandasaspdfrometna.datasets.tsdatasetimportTSDatasetfrometna.modelsimportProphetModel# Read the datadf=pd.read_csv("examples/data/example_dataset.csv")# Create a TSDatasetdf=TSDataset.to_dataset(df)ts=TSDataset(df,freq="D")# Choose a horizonHORIZON=8# Fit the modelmodel=ProphetModel()model.fit(ts)# Make the forecastfuture_ts=ts.make_future(HORIZON)forecast_ts=model.forecast(future_ts)TutorialsWe have also prepared a set of tutorials for an easy introduction:NotebookInteractive launchGet startedBacktestEDAOutliersClusteringDeep learning modelsEnsemblesDocumentationETNA documentation is availablehere.AcknowledgmentsETNA.TeamAlekseev Andrey,Shenshina Julia,Gabdushev Martin,Kolesnikov Sergey,Bunin Dmitriy,Chikov Aleksandr,Barinov Nikita,Romantsov Nikolay,Makhin Artem,Denisov Vladislav,Mitskovets Ivan,Munirova AlbinaETNA.ContributorsLevashov Artem,Podkidyshev AlekseyLicenseFeel free to use our library in your commercial and private applications.ETNA is covered byApache 2.0.
Read more about this licensehere
Package: etnotify
ETNA Notification watcher.
Gets informations about the latest validations.InstallThis project won’t work inside a virtualenv because of the dbus dependency.$gitclonehttps://github.com/tbobm/etnotify$cdetnotify$pipinstall-rrequirements.txt$pythonetnotify.pyConfiguration$exportETNA_USER=yourlogin$exportETNA_PASS=yourpasswordAim of the EtnotifyEver felt like something important on the intranet was missing ? Like notifications?This script aims to fix that, by allowing you to be alarmed when something new happens.In a rush ?You’ll get notified if your validation requests have been accepted or rejected, in the blink of an eye.
Package: etools-offline
eTools Offline Collect makes use ofunicef_attachmentsto handle the files.Installation$ pip install etools-offlineSetupAddetools_offlinetoINSTALLED_APPSin settingsINSTALLED_APPS = (
  ...
  'etools_offline',

)Set eTools Offline API settings;ETOOLS_OFFLINE_API = "http://localhost:8000/api/remote/blueprint/"
ETOOLS_OFFLINE_TOKEN = "123"UsageHooks happen in serializers, so update relevant serializers;from etools_offline import OfflineCollect

# add new blueprint
OfflineCollect().add(data={
    "code": "blueprint-123",
    "form_title": "Blueprint 123",
    "form_instructions": json.dumps({"key": "value"}),
    "accessible_by": ["joe@example.com", "mary@example.com"],
    "api_response_url": "http://example.com/response/",
})

response = {
    'id': 88,
    'code':
    'blueprint-123',
    'form_title': 'Blueprint 123',
    'form_instructions': {'key': 'value'},
    'is_active': False,
    'accessible_by': ['joe@example.com', 'mary@example.com'],
    'expiry_date': '2021-02-18',
    'max_forms_allowed': None,
    'api_response_url': 'http://example.com/response/',
    'created': '2020-02-19T15:41:10.016068Z',
    'modified': '2020-02-19T15:41:10.016094Z'
}

# update blueprint
OfflineCollect().update(
  accessible_by="<accessible_by>",
  expiry_date="<expiry_date>",
)

# get list of blueprints
OfflineCollect().list()

# get specific blueprint
OfflineCollect().get(<code>)

# delete blueprint
OfflineCollect().delete(<code>)ContributingConfigure development environment$ make developRun tests$ make test
Package: etools-validator
Validator is a library that provides an improved handling of validation.The following parts of validation are handled;statetransistionInstallationpipinstalletools-validatorSetupAddvalidatortoINSTALLED_APPSinsettings.py.. code-block:: pythonINSTALLED_APPS = (…
‘etools_validator’,)UsageUsevalidatorin viewsfromvalidator.mixinsimportValidatorViewMixinclassExampleView(ValidatorViewMixin,ListCreateAPIView):...ContributingEnvironment SetupTo install the necessary libraries$ pip install -r requirements/dev.txtCoding StandardsSeePEP 8 Style Guide for Python Codefor complete details on the coding standards.To run checks on the code to ensure code is in compliance$ flake8 .TestingTesting is important and tests are located intests/directory and can be run with;$ coverage run manage.py testCoverage report is viewable inbuild/coveragedirectory, and can be generated with;$ coverage htmlLinksStableDevelopmentSource Codehttps://github.com/unicef/etools-validatorIssue trackerhttps://github.com/unicef/etools-validator/issuesCompatibility Matrix
Package: etos-client
ETOS (Eiffel Test Orchestration System) ClientDescriptionETOS Client is the client tool for executing test suites within ETOS.FeaturesEasy to get started with.Powerful integration with ETOS.Interacts well with automation server.InstallationInstall the project by running:pip install etos_clientExamplesetos_client -i pkg:docker/etos/suite_runner@1.0.0 -s http://mytestsuite.jsonContributeIssue Tracker:https://github.com/eiffel-community/etos/issuesSource Code:https://github.com/eiffel-community/etos-clientSupportIf you are having issues, please let us know.
There is a mailing list at:etos-maintainers@googlegroups.comor just write an Issue.
Package: etos-lib
ETOS (Eiffel Test Orchestration System) LibraryDescriptionA collection of common tools and code used throughout the whole ETOS project.Installationpip install etos_libContributeIssue Tracker:https://github.com/eiffel-community/etos/issuesSource Code:https://github.com/eiffel-community/etos-librarySupportIf you are having issues, please let us know.
There is a mailing list at:etos-maintainers@googlegroups.comor just write an Issue.
Package: etos-test-runner
ETOS (Eiffel Test Orchestration System) Test RunnerDescriptionThe ETOS Test Runner is responsible for executing test suites in ETOS.
ETR should, ideally, be executed within a test runner docker container.Installationpip install etos_test_runnerContributeIssue Tracker:https://github.com/eiffel-community/etos/issuesSource Code:https://github.com/eiffel-community/etos-test-runnerSupportIf you are having issues, please let us know.
There is a mailing list at:etos-maintainers@googlegroups.comor just write an Issue.