Position: Data Engineer
Company: Upstaff
Category: Data Engineer
Experience required: 3y
English level: upper

Summary
- 5+ years of experience in data engineering with Python; 
- Strong Data modeling skills;
- 2+ years of working with data warehouses like Snowflake / BigQuery / Redshift;
- Strong experience in writing complex SQL;
- Strong experience in writing ETL / ELT data pipelines;
- Experience with data flow automation tools like Airflow / Prefect/ Dagster CI/CD processes/ Jenkins; 
- Experience at least 1 cloud AWS, GCP, or Azure;
- Good understanding of distributed systems; 
- Experience with DBT;
- Experience with Snowpark API;
- Experience with big data processing frameworks (PySpark, Dask, etc.);
- Experience with Kubernetes / Docker.

Responsibilities and Requirements
- Our customer within the energy domain is looking for 2 Data Scientists to join the team;
- In this project, the consultant will play a crucial role in migrating existing R-based machine learning models from the older platform to Azure Machine Learning;
- The consultant will be responsible for adapting, training, and testing the models on a new platform;
- Furthermore, they’ll be in charge of ensuring the functionality and performance of the new platform;
- The consultant should have previous experience working with Azure Machine Learning, R, Azure DevOps, and other Azure Services.


Duties and Responsibilities 
- Architect and implement effective and robust data solutions;
- Make clever engineering and product decisions based on data analysis and collaboration.

Required skills 
- 5+ years of experience in data engineering with Python; 
- Strong Data modeling skills;
- 2+ years of working with data warehouses like Snowflake / BigQuery / Redshift;
- Strong experience in writing complex SQL;
- Strong experience in writing ETL / ELT data pipelines;
- Experience with data flow automation tools like Airflow / Prefect/ Dagster CI/CD processes (GitLab CI/CD, GitHub CI/CD, Jenkins); 
- At least 1 cloud (AWS / GCP / Azure) data engineering stack and experience implementing effective solutions;
- Good understanding of distributed systems; 
- Experience with data quality processes, checks, validations, the definition of data quality metrics, and measurement.

Education and degree: 
- Master's or Bachelor's Degree in a quantitative field (Math, Statistics, Computer Science, Engineering, Data Science, Operations Research, etc.). 

Will be a huge plus: 
- Experience with DBT;
- Experience with Snowpark API;
- Experience with big data processing frameworks (PySpark, Dask, etc.);
- Experience with Kubernetes / Docker.

Skills and abilities: 
- Team playing result in orientation skills Self-management skills