Position: Senior/Principal Data Engineer (Python, Spark, Databricks, Azure)
Company: GlobalLogic
Experience required: 5y
Category: Data Engineer
English level: upper
Published: 2022-12-01T00:00:00+02:00

Our client is one of the leaders in Software as a service in the world of business applications.

Description:

Our client is one of the leaders in Software as a service in the world of business applications.
You will be working on a data platform.

About the platform:
A suite of products to deliver a flexible modern data estate in the customer’s Azure tenant without the major expense and effort of a large buildout or data program.
We build, deliver, and update a data platform for customers to build on top of, without the hassle and expense of implementation.
Customers see data available in their environment in week one, with sophisticated history and query technology – no engineering!

Requirements:

- 4+ years building conceptual, logical, and/or physical database designs, or the pipelines required to build them
- 2+ years of hands-on Python experience (especially WHL and packed code)
- Expert-level knowledge of using SQL to write complex, highly optimized queries across large volumes of data.
- 2+ years of Kimball (Dimensional Modeling) Expertise
- 2+ years of Spark experience (especially Databricks Spark and Delta Lake)
- Experience with source control (git) on the command line
- Strong hands-on experience implementing big-data solutions in the Azure Data Lake ecosystem (Azure Data Lake, Databricks).
- Ability to work independently and provide guidance to junior data engineers.
- (preferred) Experience with big data pipelines or DAG Tools (Dbt, Data Factory, Airflow, or similar)
- (preferred) Experience with Kafka or other streaming technology (or a willingness to learn)
- (preferred) Experience with database deployment pipelines (i.e., dacpac’s or similar technology)
- (preferred) Once or more unit testing or data quality frameworks
- (bonus) Experience with MLFlow and other MLOps pipeline technology

Preferences:
- azure data factory, Kafka

Job Responsibilities:

- Take a major part in requirements definition, architecture, and design;
- Architect of subsystems and understand the impact across products;
- Design and implement core data flow components in accordance with project standards;
- Regular communications with the client and teams;
- Ensure project delivery in a timely matter;
- Lead engineering efficiency efforts. Improve operational efficiency;
- Daily project activities (daily standups, technical advisory, reporting, etc);
- Collaborating with the Dev Leads and Product Manager to ensure that the requirements;
- Managing daily project activities.