Position: Data Engineer, Online Marketplace, Sign-on Bonus
Company: DataArt
Experience required: 2y
Category: Data Science
English level: upper
Published: 2021-06-01T00:00:00+03:00

We offer a $1000 sign-on bonus to the specialist that will accept our offer and join DataArt in this position. $1000 will be included in the first salary.

Responsibilities:

Due to a legal requirement the client has a task to migrate all their images to another storage (currently all the client's images are stored in an S3 bucket and they take up around 400TB)
Applying governance over data – switching from Amazon internal messaging system to Kafka
Building streaming platform based on manage service from Amazon
Work on different levels – data knowledge, platform knowledge, infrastructure knowledge
Clean up conversations that are linked to some groups of users
Data cleanup details: Image migration – images are currently stored in S3 bucket and take up to 400 TB – they need to be moved to new bucket and deleted from old.
Akamai needs to be reconfigured to serve from new bucket correctly
Chat database cleanup – research stage on how to approach the deletion, very big database, load tests

Must have:

Hands-on experience in data engineering (e.g., working with large volumes of data, cloud environments, parallelization)
Knowledge of AWS technologies (e.g., S3, Aurora)
Akamai configuration would be a huge plus
Spoken English, good communication and coordination skills