Package: pyblux
pyblux: A suite of fast, easy-to-use, and intuitive Python ETL utilities.What is it?pybluxis a Python package that provides a suite of ETL utilities built to make the interactions with databases in the cloud as well as on-premise fast, easy and intuitive.FeaturesSupport multiple databases, including Postgres, MySql, MS SQL, SQLIte, Teradata and Oracle.Theget_enginemethod makes it easy to connect to databases in a simple and intuitive manner.Blux.sqlmethod from theBluxclass helps run fast queries. It Povides output results as namedtuple or dictionary and it supports parameterised queries and in-flight transformation of data.Loggerclass helps setup logging via log file or console.send_teams_notificationmethod provides an easy way to send alerts to a MS Teams channel via anincoming webhook.send_emailmethod helps to send email with HTML contentHelpful error messages display the failed query SQLDBAPI2 specificationwas used in order to simplify coding for queries on relational database systems using Python.Where to get itThe source code is currently hosted on GitHub at:https://github.com/bertin.nono/pybluxBinary installers for the latest released version are available at thePython
Package Index (PyPI)# or PyPIpipinstallpybluxpyblux provides support for the databases below:TeradataPostgreSQLMySQL and MariaDBSQLiteOracleMicrosoft SQL ServerDependenciesDepending on the use case, the database package should be installed.Teradata:  Install the Teradata SQL Driver Dialect for SQLAlchemypip install teradatasqlalchemyPostgreSQL: Install Psycopg which is the most popular PostgreSQL database adapter for the Python programming language.pip install psycopg2-binaryMySQL and MariaDB: Install the Pure Python MySQL Driverpip install PyMySQL[SQLite]: No install requiredOracle: Intall cx_Oracle which is a Python extension module that enables access to Oracle Database.pip install cx-OracleMicrosoft SQL Server: Install the pyodbc which is an open source Python module that makes accessing ODBC databases simple.pip install pyodbcDocumentationClassesMethodsReferencesClassesBlux:Establishes a connection engine to a database system referenced by the dialect attribute run fast queries.Note:Thedialectis the system SQLAlchemy uses to communicate with various types ofDBAPIimplementations and databases.class Blux:
    """
    This class connects to a local database session using the db `dialect` library.
    """

    def __init__(self, engine=None,dialect=None):
        """
        Args:
            engine (str): Database connection engine.
            dialect (str): database system name(postgres, oracle, teradata,...)
        Note: Database Connection package must be installed in order to use this backend.
        """
        self.engine = engine
        self._dialect = dialect

        self.__errlimit = 1
        self.__warnings = []
        self.__errors = []
        self.__logons = []Example:table_list="""SELECT table_schema, table_name FROM information_schema.tables"""postgres_engine=get_engine(dialect='PG',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")blux=Blux(engine=postgres_engine,dialect='postgres')dataframe=blux.sql(query=table_list)table='test'database='postgres'#load dataframe to tableblux.sql(dataframe=dataframe,database=database,table=table,dialect='postgres')Logger:provides a custom logging handler calledlogger. Helps Debug SQL and monitor progress with logging.classLogger:"""This class connects to a local database session using the db `connnection` library."""def__init__(self,logname:str,filename:str,level=logging.INFO,console:bool=True):"""Args:logname (str): Logger Name.filename (str): log file pathlevel (str): Logger Level (DEBUG, INFO, WARNING, ERROR)console (cool): print to console"""self._logname=lognameself._filename=filenameself._level=levelself._console=consoleExample:importloggingfrompybluximportloggerpyblux_logger=Logger(logname=ETL.NAME,filename=log_file,level=logging.INFO,console=True)logger=pyblux_logger.logger(verbose=True)Output from a call forget_enginewill look like:2021-07-07 15:06:22,411 get_engine: 
2021-07-07 15:06:22,413 get_engine: 
2021-07-07 15:06:22,416 get_engine:Methods:get_engine:Creates a database connection engine.get_engine(user:str,password:str,host:str,port:int,database:str,dialect:str,verbose:bool=False,parameter:str=None,raw_engine:bool=True,logger:Callable=print)Database connection details are defined byget_engineobjects (see below).Example:importpandasfrompyblux.utilsimportget_enginefrompyblux.bluximportBluxoracle_engine=get_engine(dialect='oracle',host="localhost",port=1521,database="mydata",user="oracle_user",password="123")teradata_engine=get_engine(dialect='mssql',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_engine=get_engine(dialect='mysql',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_engine=get_engine(dialect='teradata',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_engine=get_engine(dialect='postgres',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")PasswordsIt is best practice for Database passwords to be stored in environment variables.
This can be done on the command line via:export password=secret-passwordon Linuxset password=secret-passwordon WindowsOr in a Python terminal via:importosos.environ['password']='secret-password'No password is required for SQLite databases.When URL that includes the password containsPassword conatains special charactersExample:Connection_String = postgresql+psycopg2://user:p@ssword%to%encode@hosturl/defaultdb
The above password encoded using urllib.parse:pwd=urllib.parse.quote_plus("p@ssword%to%encode")
   print(pwd)
   Connection_String = postgresql+psycopg2://user:urllib.parse.quote_plus("p@ssword%to%encode")@hosturl/defaultdbget_connection:Gets a regular database connection.get_connection(user:str,password:str,host:str,port:int,database:str,dialect:str,verbose:bool=False,parameter:str=None,logger:Callable=print):
    """
    Get a regular connection for Teradata , Oracle, Aurora/Postgres, Aurora/MySql/MariaDB, SQLite, and  Microsoft SQL Server
    Returns
    -------
    Connection Object
    """Example:frompyblux.utilsimportget_connectionoracle_conn=get_connection(dialect='oracle',host="localhost",port=1521,database="mydata",user="oracle_user",password="123")teradata_conn=get_connection(dialect='mssql',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_conn=get_connection(dialect='mysql',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_conn=get_connection(dialect='teradata',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")postgres_conn=get_connection(dialect='postgres',host="localhost",port=5432,database="mydata",user="postgres_user",password="123")create_table_from_dataframe:Creates table from a dataframe attributes and fastload data into in it.create_table_from_dataframe(dataframe:pd.DataFrame=None,table:str=None,Blux:Blux=None,verbose:bool=False,logger:Callable=print)is_exist:Checks is a table or view exist.is_exist(table:str='', Blux:Blux=None,verbose:bool=False,logger:Callable=print)drop_table:Checks is a table or view exist and then drops it if it exists.drop_table(table:str=None,Blux:Blux=None,verbose:bool=False,logger:Callable=print)send_teams_notifications:Send a Card to a MS Teams Channelsend_teams_notification ( hookurl: str, title: str='' , text: str='', message: str ='', status: str ='', error_message: str='', activitySubtitle: str='', activityText: str='')send_email:Send an HTML formated email that can include a dataframesend_email(server:str, port:int,sender: str, receivers: list, subject: str, body_text: str, attachment: any = None,df: pd.DataFrame = None)Maintainers:Bertin NonoDevelopment statusStableLicenceMITReferencespsycopg2cx_OraclesqlalchemyChange Log(09/12/2021)Initial Release