Position: Lead Data Platform Engineer
Company: thredup
Category: Scala
Experience required: 5y
English level: upper
Published: 2020-10-01T00:00:00+03:00

At thredUP, we live a true data driven culture with an ever growing appetite for data and a mindset to generate insights and make informed business decisions. In this role, you will shape the roadmap for the next generation of data-platform that is easy to use, elastic, and cost-efficient.

This platform should make it easy for data engineers to build batch and streaming pipelines as well as make it easy to interact with the data for the end consumers. Build a  robust machine-learning platform to develop, test, and deploy machine learning models deployed for both batch and real-time use cases. 
Additionally, support the current technology stack and evaluate newer tools/technologies that reduce the support cost and provide improved productivity and experience.

Responsibilities: 

Ideate and build the next generation product roadmap for the data platform team.
Own Airflow deployments, integration with Datadog and CI/CD.
Own the Mlflow machine learning platform and find alternatives or enhance it to support input data versioning and model deployment in A/B testing mode.
Build and evangelize reusable components for data pipelines.
Implement a solution to capture metadata and lineage.
Own the enterprise event bus solution.
Own the databricks environment.
Own and shape the feature store infrastructure and roadmap.

Requirements: 

8+ years of experience in building scalable data platforms and tools.
Demonstrated experience in implementing Spark for data processing.
Demonstrated experience in implementing Kafka for real time data processing.
Demonstrated experience providing rest endpoints to expose data to other applications.
Experience  integrating and supporting Databricks.
Working with noSQL stores like HBASE/DynamoDB.
Experience providing solutions to handle data privacy.
Experience in implementing machine learning platforms like MLFlow, Tensorflow, Sagemake.
Experience in deploying Airflow as a scheduling tool.
Experience in implementing CI/CD pipelines using Jenkins.
Expert level using Python and PySpark.
Understanding with AWS Data technologies (such as Redshift, S3, Glue)
Experience with repositories like Git, maven, jfrog.
Excellent communication and problem solving skills.
Well organized and able to manage multiple projects in a fast-paced demanding environment


What We Offer:
High competitive top market salary
The opportunity to make a massive impact & influence outcomes for our business and customers alongside passionate coworkers
Autonomy. The ability to make, own, and carry out decisions
Working within a modern tech stack
Flexible working hours (possibility to work from home on Tuesdays and Thursdays)
Uncounted paid vacations
Full-coverage medical insurance, free lunches on Wednesdays, English classes, etc.
Relocation assistance and cost coverage program for candidates from other countries and cities