Position: DataOps Engineer
Company: IT-Rex
Experience required: 1y
Category: Data Engineer
English level: upper
Published: 2023-01-01T00:00:00+02:00

Datawarehouse Luxembourg builds, maintain and operates applications related to Clearstream Data Warehouse area via numerous applications called: CDL application (Clearstream Data Layer built on cloud latest technologies), Claims and Risk Management, Risk management, Regulatory Reporting

Middle

experience with cloud native technologies;
experience with big data, data architecture - Data Factory, Data Fusion, or Apache Airflow;
experience with Python, CI/CD, Jenkins.

Requirements:
Hands on experience with cloud native technologies, Azure/GCP;
Direct experience in building data pipelines such as Data Factory, Data Fusion, or Apache Airflow;
Demonstrated experience in one or more programming languages, preferably Python;
Good knowledge of CI/CD and version control tools such as Jenkins and GitHub Actions;
Very strong English language skills are a must;

Familiarity with data platforms and technologies such as Kafka, Delta Lake, Spark; Databricks is a plus.

Your responsibilities:

Design, build and optimize data pipelines to facilitate the extraction of data from multiple sources;
Be first level support for end user’s requests when dealing with data pipeline issues;
Develop ETL/ELT (batch/stream) from multiple sources using Spark and/or Kafka;
Operate the data pipelines to ensure key SLAs are managed across a wide range of producers and consumers;
Support various components of the data pipelines, including ingestion, validation, cleansing and curation;
Promote data collaboration, orchestration, quality, security, access, and ease of use;
Gather data requirements from analytics and business departments;
Write and maintain operational and technical documentation and perform tasks in Agile methodology).