Position: Big Data Engineer
Company: AppReal-VR.COM
Experience required: 3y
Category: Scala
English level: upper
Published: 2020-11-01T00:00:00+02:00

The ideal candidate should be able to use his experience in implementing advanced data streaming applications that ingest ~10TB daily data which feeds all of the organization units including marketing, BI, AI and statistical/mathematical models.
You will working with professional data teams built from data engineers and architects, data scientists, economy managers & Business analysts.
Responsibilities:
• Architect, Design and implement ETL pipelines for multiple data sources ingesting massive amounts of data
• Tune and optimize data applications architecture to reach real-time capabilities.
• Promote best practices coding, code complete standards, OO programing & design patterns
• Develop smoke and integration tests with a running a daily build based CI
• Champion the overall strategy for data governance, security, and quality that ensure requirements are met.
• Collaborating with our architects to capture, format and prepare data for machine learning train and test
Requirements:
• Proficient in Java with +3 years of experience in writing Spark based batch and streaming data applications (Streaming\Structured Streaming\SparkSQL)
• Experience integrating at least one Big-Data Key\Value store (Aerospike, Redis, DynamoDb, MongoDb, Hazalcast, Cassandra, HBase, Memcached)
• Strong understanding of the Hadoop ecosystems (especially Kafka, hdfs, Zookeeper, Yarn, Mesos, Parquet, Hive) and related technologies with +2 years of experience
• Strong working knowledge of SQL or SQL-like data management languages with +5 years of experience
• Linux knowledge with bash scripting experience
• Deep understanding of strong Computer Science fundamentals: object-oriented design using best practices and design patterns, data structures systems, applications and multithreading programming 
• Passion for data engineering automation and efficiency, self-starter, loves challenges, independent and a tech savvy
• Advantage: knowledge in K8s, Docker , DeltaLake, Aerospike Redis, Vertica
• BSc in computer science or equivalent