Position: Big Data Engineer (Hadoop, Spark, Scala), Investment Management Solution
Company: DataArt
Experience required: 3y
Category: Data Engineer
English level: intermediate
Published: 2022-09-01T00:00:00+03:00

CLIENT
Our client is one of the world’s top 10 investment companies with headquarters in Great Britain with branch offices in the US and Hong Kong.

PROJECT OVERVIEW
The project is a Single Source of Truth for investment management data and investment analysis solution that will aggregate and process data from multiple data feeds and systems like stock exchanges, news feeds, stock exchange brokers, internal quantitative systems and others. The development is just started and it's a great chance to join the team from the very beginning.

TEAM
The project involves around 90 various specialists (30 Big Data engineers, 10 data and business analysts, InterSystems cache developers, Java developers, and Quantitative analysts/developers, etc.).

POSITION OVERVIEW
As a Big Data Engineer, you will be developing a platform that models and predicts trends in the financial markets.

TECHNOLOGY STACK
Hadoop/Cloudera, Spark, Scala.

Responsibilities:
- Manage existing and create new Spark jobs (Scala)
- Optimizing and performance tuning of Spark jobs
- Creating and update NiFi pipelines, manage huge NiFi environment
- Manage Hive/Impala tables in CDH infrastructure
- Support during creation and implementation of the architecture
- Data analysis
- Participation in Scrum ceremonies
- Interaction with BA

Requirements:
- Technical expertise with Big Data tools and methodologies
- Hands-on experience with Spark 2.0 or greater (+ Scala)
- Programming experience with Scala
- Experience working with Hadoop ecosystem (CDP/Cloudera Data Platform)
- Proficiency in SQL
- Spoken English

Nice to have:
- Experience working with Hive And Impala
- Experience working with Apache NiFi
- Experience working with Kafka
- Experience working in an agile environment and understanding of various phases of the software development life cycle
- Experience with Data modelling
- Skills working with Airflow (preferred) or Oozie