Position: Data Engineer with Scala and Python, Customer Intelligence Platform
Company: DataArt
Experience required: 2y
Category: Data Engineer
English level: intermediate
Published: 2022-03-01T00:00:00+02:00

CLIENT:
DataArt’s client is a UK-based customer intelligence platform focusing on the telecommunications industry. The platform was developed to handle data at this enormous scale efficiently and allows to enable the marketing personalization thanks to the cutting-edge machine learning. Our customer explores and exploits trillions of rows of data to provide the best possible insights about how millions of users behave, how they think and who they are. Security, privacy and data ethics are at the core of the platform.

TECHNOLOGY STACK:
Amazon Web Services, Python, Scala. • Scala to develop most of the backend code, including REST services, data pipelines, making use of FS2 functional streams data processes, Cats Effect, IO monad, Circe, Doobie, Http4s and other Typelevel stack libraries; • Kafka, SQS as distributed messages systems; • GitHub platform for a source code repository, CI/CD, tasks tracking; • Cassandra, PostgreSQL, Elasticsearch, S3, HDFS as the data stores; • Docker, Nomad, Terraform, Consul and Vault in the operational stack; • Apache Spark park for transforming multi-terabyte sets of data into valuable insights.

Responsibilities:
- Product development, new data sources integrations and pipelines creation.

Requirements:
- 2+ years of Scala or Java 8+ functional programming experience.
- Skills working with Apache Cassandra.
- Experience working with Kafka.
- Experience working with PostgreSQL database.
- Strong knowledge of Spark.
- Experience with microservices architectures and testing practices.
- Good understanding of distributed architecture applications.
- Experience with various CI/CD tools.
- You’re comfortable working in Agile/Lean environments.
- Willingness to learn and share knowledge with your teammates.
- Good spoken English and strong communication skills.