Position: Big Data Engineer (Hadoop, StreamSets), Banking System
Company: DataArt
Experience required: 2y
Category: Data Science
English level: intermediate
Published: 2021-04-01T00:00:00+03:00

**About the vacancy**

Our client is a British bank, one of the largest financial services companies in the world. The bank was founded three centuries ago, and today its assets are estimated to be worth trillions of US dollars.

DataArt's experts together with developers on the client's side are creating new banking apps and upgrading the old ones.

The Data Engineer will be a part of the data engineering community to deliver opportunities to support the organization’s strategic direction. The candidate will be working primarily with business analysts and a test manager.

The specialist will use Data Lake, Data Science, and Streaming technologies to help shape the data flow and architecture of large programs of work like Customer Due Diligence.

Tech stack and Infrastructure: StreamSets, Spark, Hadoop, Python, SQL, BigData (DWH).

**Responsibilities**
- Systems analysis
- Data analysis
- StreamSets pipeline design
- Work with documentation
- Testing of pipelines
- StreamSets pipeline construction/development
- Change and configuration management of delivered components

**Must have**
- Strong knowledge of Big Data general concepts
- Strong knowledge of Java/Scala or Python
- Experience working with Hadoop Ecosystem (HDFS, Map-Reduce, Yarn, Hive)
- Knowledge of NoSQL DB (any popular)
- Understanding of Spark (Core, SQL, Streaming)
- Strong Knowledge of Kafka
- Experience with StreamSets
- Spoken English
**
Would be a plus**
- Knowledge of Informatica ETL
- Knowledge of Cloudera
- Experience with Public Clouds (AWS, Azure, Google, etc.)
- DevOps skills
- Experience troubleshooting production systems