Position: Big Data/Data Engineer, Investment Management Solution
Company: DataArt
Category: Data Engineer
Experience required: 3y
English level: intermediate

**About the vacancy**

Our client is one of the world’s top 10 investment companies with headquarters in Great Britain. They are building a data market that will aggregate and process data from multiple data feeds and systems like stock exchanges, news feeds, stock exchange brokers, internal quantitative systems and others. The project involves around 90 various specialists (30 Big Data engineers, 10 data and business analysts, InterSystems cache developers, Java developers and Quantitative analysts/developers, etc.).

As a big data engineer, you will be developing a platform that models and predicts trends in the financial markets.

**Must have**
- Experience working with Apache NiFi or Hadoop (HDFS, MapReduce, HBase)
- Hands-on experience with Shell scripting
- Strong experience with SQL
- Spoken English

**Would be a plus**
- Experience working in an agile environment and understanding of various phases of the software development life cycle
- Hands-on experience with Spark 2.0 or greater (+ Scala)
- Experience working with Hive And Impala
- Experience with Data modelling
- Skills working with Oozie
- Skills working with Airflow
- Skills working with Datasets and Data frames
- Skills using source code and version control systems like SVN, Git, etc.
- Experience on CDP(Cloudera Data Platform)
- Knowledge of Hadoop architecture components