Position: Big Data Engineer
Company: NCube
Experience required: 3y
Category: Python
English level: upper
Published: 2020-04-01T00:00:00+03:00

Experience / Skills:

Must have:

Experience as a software engineer
Experience with AWS/Cloud Big Data computing design, provisioning, and tuning
Knowledge of database concepts, object and data modeling techniques and design principles
Detailed knowledge of database architectures, software, and facilities
Successful history of manipulating, processing, and extracting value from large disconnected data sets
Experience with programming languages — Python (required) and/or Scala and/or Ruby and/or R
Database technologies — SQL, performance tuning concepts, AWS RDS, RedShift, MySQL
Experience with big data batch processing tools: Hadoop MapReduce, ElasticSearch, PIG, Hive, Cascading/Scalding, Apache Spark, AWS EMR
Experience with stream-processing systems: Kinesis, Kafka, MQTT
Experience with relational NoSQL databases including DynamoDB

Will be a plus: 

Bachelor’s degree in computer science, computer engineering, or a related field, or the equivalent combination of education and related experience
Related AWS certification, preferred
Previous experience as a Data Engineer / Database Administrator and/or Business Intelligence Analyst
Ability to write JSON, XML, YAML and other data definition schemas

We offer: 

Excellent career opportunities
Competitive remuneration based on qualification and contribution
Flexible working schedule, vacations, paid sick leaves
Good working environment
Great team spirit
Perfect office location in the very center of Kyiv

Responsibilities: 

Build distributed, scalable, and reliable data pipelines that ingest and process data at scale and in real-time
Provide solutions for the design and implementation of Hadoop EMR Cluster/ Big Data Infrastructure
Deploy and maintain Hadoop/Big Data/Spark and database storage Infrastructures in AWS cloud
Create data environments and/or data sets to serve a wide range of data users, including but not limited to Data Scientists, Data Analysts, Business Analysts etc
Perform offline analysis of large data sets using components of a big data software ecosystem
Support the evaluation of big data technologies and prototype solutions to improve data processing architecture
Troubleshoot and determine root cause of complex data provenance, metadata issues and engineering questions that may involve interfacing with various technical staff in multiple organizations and with differing levels of expertise
Investigate, evaluate, test and recommend technical solutions for future systems