Position: Data Engineer
Company: GoInteractive
Experience required: 5y
Category: Data Engineer
English level: upper
Published: 2022-03-01T00:00:00+02:00

On this project, you will (along with our team) help develop a cutting edge system which generates efficient idiomatic PySpark and Pandas code for Data Engineering use cases. This code will be used to handle a wide variety of data types from multiple sources, performing transformations according to the user’s requirements.

In addition to PySpark/Pandas for pure Data Engineering, the system will also have uses in generating scripts for Machine Learning / Data Science applications. Prior Machine Learning or Data Science knowledge is a plus, but not necessary to perform the role.

You will use both core object oriented software engineering design principles as well as scripting techniques to delve into difficult code generation problems, finding solutions that solve important issues around readability and performance. Your knowledge of PySpark and Pandas will help you design solutions that can be assembled automatically into larger programs that perform complex operations. The role will give you the opportunity to contribute ideas and work creatively, with the opportunity to extend the engagement further.

Responsibilities:
- Develop production ready software in a collaborative manner with other expert-level software engineers
- Utilize best practices including giving and receiving code reviews, architectural reviews and over-communicating about risks related to estimates and planning
- Contribute expertise with Python, PySpark and Pandas to the project team
- Use Git, Github, CI/CD, Pytest, and technical specifications to coordinate high quality technical work
- Write down technical requirements and collaboratively solve problems, managing risk and schedule
- Bring positive energy and a can-do attitude to talented team of similarly positive and motivated engineers
- Communicate early and often in both spoken and written English over platforms like Slack, Email and video calls.

Requirements:
- 5+ years of experience with Python
- Experience with PySpark and Pandas in production environment
- Experience in design and implementation of data processing pipelines
- Working knowledge of Spark Python API
- Experience with performance tuning and monitoring of Spark jobs
- Apache Spark SQL

We offer:
- Competitive compensation, based on experience
- A culture dedicated to professional growth, with continuing education enrichment
- Possibility to work remotely
- Medical insurance
- An open, transparent and fun work culture
- Career and professional growth
- Long-term employment with 20 working days paid vacation and other social benefits
- Modern and comfortable office very close to the subway