Package: airflow-indexima
airflow-indeximaVersions followingSemantic VersioningOverviewIndeximaAirflowintegration based on pyhive.This project is used in our prod environment with success.
As it a young project, take care of change, any help is welcome :)SetupRequirementsPython 3.6+InstallationInstall this library directly into an activated virtual environment:$ pip install airflow-indeximaor add it to yourPoetryproject:$ poetry add airflow-indeximaor you could use it as anAirflow pluginUsageAfter installation, the package can imported:$ python
>>> import airflow_indexima
>>> airflow_indexima.__version__SeeApi documentationa simple queryfromairflow_indexima.operatorsimportIndeximaQueryRunnerOperator...withdag:...op=IndeximaQueryRunnerOperator(task_id='my-task-id',sql_query='DELETE FROM Client WHERE GRPD = 1',indexima_conn_id='my-indexima-connection')...a load into indeximafromairflow_indexima.operators.indeximaimportIndeximaLoadDataOperator...withdag:...op=IndeximaLoadDataOperator(task_id='my-task-id',indexima_conn_id='my-indexima-connection',target_table='Client',source_select_query='select * from dsi.client',truncate=True,load_path_uri='jdbc:redshift://my-private-instance.com:5439/db_client?ssl=true&user=airflow-user&password=XXXXXXXX')...get load path uri from ConnectionIn order to get jdbc uri from an Airflow Connection, you could use:get_redshift_load_path_uriget_postgresql_load_path_urifrom moduleairflow_indexima.uriBoth method have this profile:Callable[[str, Optional[ConnectionDecorator]], str]Example:get_postgresql_load_path_uri(connection_id='my_conn')
    >> 'jdbc:postgresql://my-db:5432/db_client?ssl=true&user=airflow-user&password=XXXXXXXX'Indexima ConnectionAuthenticationPyHive supported authentication mode:'NONE': needs a username without password'CUSTOM': needs a username and password (default mode)'LDAP': needs a username and password'KERBEROS': need a kerberos service name'NOSASL': corresponds to hive.server2.authentication=NOSASL in hive-site.xmlConfigurationYou could set those parameters:host (str): The host to connect to.port (int): The (TCP) port to connect to.timeout_seconds ([int]): define the socket timeout in second (default None)socket_keepalive ([bool]): enable TCP keepalive, default false.auth (str): authentication modeusername ([str]): username to loginpassword ([str]): password to loginkerberos_service_name ([str]): kerberos service namehost,port,usernameandpasswordcame from airflow Connection configuration.timeout_seconds,socket_keepalive,authandkerberos_service_nameparameters can came from:attribut on Hook/Operator classAirflow Connection inextraparameter, like this:'{"auth": "CUSTOM", "timeout_seconds": 90, "socket_keepalive": true}'Setted attribut override airflow connection configuration.You could add a decorator function in order to post process Connection before usage.
This decorator will be executed after connection configuration (see next section).customize Connection credential accessIf you use another backend to store your password (like AWS SSM), you could define a decorator
and use it as a function in your dag.fromairflow.modelsimportConnectionfromairflowimportDAGfromairdlow_indexima.uriimportdefine_load_path_factory,get_redshift_load_path_uridefmy_decorator(conn:Connection)->Connection:# conn instance will be not shared, and use only on connection requestconn.password=get_ssm_parameter(param_name=f'{conn.conn_id}.{con.login}')returnconndag=DAG(dag_id='my_dag',user_defined_macros={# we define a macro get_load_path_uri'get_load_path_uri':define_load_path_factory(conn_id='my-redshift-connection',decorator=my_decorator,factory=get_redshift_load_path_uri)},...)withdag:...op=IndeximaLoadDataOperator(task_id='my-task-id',indexima_conn_id='my-indexima-connection',target_table='Client',source_select_query='select * from dsi.client',truncate=True,load_path_uri='{{ get_load_path_uri() }}')...a Connection decorator must follow this type:ConnectionDecorator = Callable[[Connection], Connection]define_load_path_factoryis a function which take:a connnection identifiera decoratorConnectionDecoratoran uri factoryUriGeneratorFactory = Callable[[str, Optional[ConnectionDecorator]], str]and return a function with no argument which can be called as a macro in dag's operator.Optional connection parametersOn each operator you could set this member:auth (Optional[str]): authentication mode (default: {'CUSTOM'})kerberos_service_name (Optional[str]): optional kerberos service nametimeout_seconds (Optional[Union[int, datetime.timedelta]]): define the socket timeout in second
(could be an int or a timedelta)socket_keepalive (Optional[bool]): enable TCP keepalive.Note:if execution_timeout is set, it will be used as default value for timeout_seconds.Production FeedbackIn production, you could have few strange behaviour like those that we have meet."TSocket read 0 bytes"You could fine this issuehttps://github.com/dropbox/PyHive/issues/240on long load query running.Try this in sequence:check your operator configuration, and settimeout_secondsmember to 3600 second for example.
You could have a different behaviour when running a dag with/without airflow context in docker container.if your facing a broken pipe, after 300s, and you have an AWS NLB V2 :
Read againnetwork-load-balancers, and focus on this:Elastic Load Balancing sets the idle timeout value for TCP flows to 350 seconds. You cannot modify this value. For TCP listeners, clients or targets can use TCP keepalive packets to reset the idle timeout. TCP keepalive packets are not supported for TLS listeners.We have tried for you the "socket_keep_alive", and it did not work at all.
Our solution was to remove our NLB and use a simple dns A field on indexima master."utf-8" or could not read byte ...Be very welcome to add{ "serialization.encoding": "utf-8"}in hive_configuration member of IndeximaHook.This setting is set in IndeximaHook.init, may you override it ?Playing Airflow without Airflow ServerWhen I was trying many little things and deals with hive stuff, i wrote a single script that help me a lot.Feel free to use it (or not) to set your dag by yourself:importosimportdatetimefromairflow.hooks.base_hookimportBaseHookfromairflowimportDAGfromairflow_indexima.operators.indeximaimportIndeximaLoadDataOperator# here we create our Airflow Connectionos.environ['AIRFLOW_CONN_INDEXIMA_ID']='hive://my-user:my-password@my-server:10000/default'conn=BaseHook.get_connection('indexima_id')dag=DAG(dag_id='my_dag',default_args={'start_date':datetime.datetime(year=2019,month=12,day=1),'depends_on_past':False,'email_on_failure':False,'email':[],},)withdag:load_operator=IndeximaLoadDataOperator(task_id='my_task',indexima_conn_id='indexima_id',target_table='my_table',source_select_query=("select * from source_table where ""creation_date_tms between '2019-11-30T00:00:00+00:00' and '2019-11-30T12:59:59.000999+00:00'"),truncate=True,truncate_sql=("DELETE FROM my_table WHERE ""creation_date_tms between '2019-11-30T00:00:00+00:00' and '2019-11-30T12:59:59.000999+00:00'"),load_path_uri='jdbc:postgresql://myserver:5439/db_common?user=etl_user&password=a_strong_password&ssl=true',retries=2,execution_timeout=datetime.timedelta(hours=3),sla=datetime.timedelta(hours=1,minutes=30),)# here we run the dagload_operator.execute(context={})delos.environ['AIRFLOW_CONN_INDEXIMA_ID']LicenseThe MIT License (MIT)ContributingSeeContributingThanksThanks to@bartosz25for his help with hive connection details...