Position: Senior Data Engineer
Company: International Development Resources
Experience required: 5y
Category: Data Engineer
English level: upper
Published: 2022-06-01T00:00:00+03:00

Our platform is evolving, so you need to be someone who can build new functionality and learn new technologies quickly, as well as be able to maintain and extend existing code to support organizational needs.
Our Data/Reports team is responsible for building out data focused solutions used by educational organizations across the globe. The team’s focus is on improving our existing reporting and data extraction capabilities by building out our new data lake/warehouse and leveraging that structure to power our products. Our vision is to make all our products data driven by leveraging ML/AI, APIs, product research, and other capabilities powered by our data solutions.  If you thrive in a fast-paced environment, working on data solutions that support millions of users daily, then you have an opportunity to join our engineering group.

Specific roles and responsibilities:

Designs data models, builds-out models in warehouse, and develops views/queries for reports 
Implement PySpark processes that leverage SQL DML and wraps them in AWS Glue
Writes AWS Lambda functions to trigger AWS Glue jobs from AWS S3 actions
Develop and monitor data processes (ETL pipelines) 
Participates in planning, analysis, and design of new data solutions and tooling for platform
Investigates, troubleshoots, and remedies performance bottlenecks in the data platform/pipelines
Learns new technologies and keeps up with both traditional and emerging best practices 
Works independently and collaborates with other members of the team to ensure successful delivery of business features
Qualifications:

5+ years of production experience in data engineering
Working knowledge of Python, PySpark, and SQL
Engineering solutions involving relational, key/value, and big data systems 
Experience operating data platforms in cloud environments, AWS preferred
Hands-on experience with Git version control system
Excellent debugging and testing skills, and interest in quickly learning new technologies
Experience with Agile methodology and software development (pull requests and code reviews)
Commitment to following security and performance best practices
Strong sense of collaboration, teamwork, and accountability
Experience building ETL pipelines in AWS, using Glue, Lambda, EMR, Redshift, etc.
Experience with designing and building scalable data solutions in AWS