Position: Senior Data Engineer
Company: Nortal
Experience required: 5y
Category: Data Engineer
Published: 2023-01-01T00:00:00+02:00

Технології: Python, PySpark, Databricks, Azure, SQL

**About the vacancy**
Nortal welcomes an experienced Senior Data Engineer to join the project team of our client – the international mining industry giant! The company is re-imagining and digitalizing the mining process, using the latest and most innovative IT technologies, to create sustainable value across an integrated value chain.

**About our client**
Our client is a global mining company with a world-class portfolio of products that are used by a billion consumers every day. Our client relies on data and cutting-edge data science to enhance their operations. They also use advanced machine learning and artificial intelligence to optimize their processes.

**About the project**
Our client is a giant corporation with 90000 employees globally. They have their own innovative IT department handling the whole cycle of development of the products used by the organization. You will work in an international expert team to develop sophisticated software solutions that have a direct impact on almost every aspect of modern life. You will use the innovative practices and the latest technologies to deliver safe and sustainable products to their customers around the world. The company is going to generate more data than ever before, and it needs to build the systems to support this and for making better decisions. As a Data Engineer you will be responsible for building and maintaining data pipelines, as well as integrating data from various sources.

**Mission**
- Design, build, and maintain data pipelines and integrations using Python, Spark, and SQL
- Develop an integrated API to allow data access and integration with other systems
- Work with Azure cloud platforms and Databricks to process and analyze data
- Collaborate with cross-functional teams to understand data needs and requirements
- Participate in the DevOps lifecycle, including code review, testing, and deployment
- Adhere to Agile methodology and practices
- Troubleshoot and resolve technical issues as they arise
- Monitor, review, and continually improve performance

**Requirements**
- Skilled in developing quality Python / PySpark based pipelines, and implementing software development best practices to ensure high-quality standards are met
- Experience with building batch or stream pipelines leverage Databricks and other Azure stream technologies
- Hands-on experience with API Development
- Experience with unstructured and spatial data types
- Good SQL knowledge
- Clear understanding of agile best practices and proactively collaborate and support the team if needed during the whole sprint cycle
- Familiar with the DevOps lifecycle
- Experience with automation tests executable via pipeline, and version controlled with test frameworks integration
- Good command of English