Position: Big Data Retraining program for middle engineers (Warszawa, Kraków, Gdańsk, Wrocław)
Company: Grid Dynamics
Experience required: 3y
Category: Other
English level: upper
Published: 2023-07-01T00:00:00+03:00

**Project description:**
Big Data retraining program for junior/middle data engineers. Big Data is one of our core-expertise for Grid Dynamics. We constantly need new team members, and eager developers to grow our data engineering team. Engineers will be working on an emulated project and literally doing the same level of tasks as our regular engineers. The program/project includes (but is not limited): 
Uploader (Go-written binary client with its backend API to transfer data from third-party into the platform)
Scheme registry (It is a metadata store with dataset schemas including schema management capabilities)
Pipeline manager (Manage ETL pipelines: CRUD and execute. Triggers ETL pipeline on upload event Pipelines metadata storage)
(Airflow DAG) data ingestion pipeline including data decryption, validation (optionally) and ingestion.
Monitor data metrics. Uploads and integrity. At least data size, num objects, uploader entity, and path.

**Client description:**
American corporation, developer of personal and tablet computers, audio players, smartphones, software and digital content.

**Details on tech stack:**
Scala, Spark, Snowflake, docker, K8s, Helm, AWS(EKS, IAM, S3, SNS, SQS, MSK etc), Airflow, PostgreSQL, Superset/Tableau.

**Requirements to the candidate:**
Knowledge of at least one of the programming languages: Python, Scala, Go, Java
Hands-on experience with data engineering projects (Clouds or in-premise) at least 2 years
Commercial experience, working with real Clients (not only pet projects or RnD or Internships)
Eager to invest his/her time in education and learn new technologies
English level - at least B2

**Current team size, their locations and whether team is going to grow?**
5-10 team members + Team Leader


**What is good about the project?**
Opportunity to work in a very experienced team with, great culture
To grow within the account (getting new projects, roles, disclosing the client's projects.)
Relocation opportunity (US in case of successful performance and good client feedback)