Position: Lead Big Data Engineer
Company: EPAM
Category: Data Engineer
Experience required: 5y
English level: upper
Published: 2023-04-01T00:00:00+03:00

**DESCRIPTION**
Our client, a world-famous, public California-based multinational technology company, specializing in consumer electronics, software, and online services, extends its partnership with EPAM.

You will work with world-class experts and technologies from the heart of Silicon Valley, developing products and services that billions of people use every day. 

**ABOUT POSITION**
We are looking for a Lead Software Data Engineer:
The position is a software engineering. It’s not a data-science or analytics position where adhoc data exploration would be primary job. We write software, data processing jobs, data pipelines for consumption by our internal and external partners. Although we write software, data is our main product and 1st class citizen. We care about correct and qualitative data that we produce as much as we care about fine, clean and manageable code.

**RESPONSIBILITIES**
- Write new data pipelines, jobs to produce new outputs (datasets) in scope of new features development
- Adopt existing data pipelines to integrate with new org-wide platforms, tools, services, languages
- Fix bugs in code and correct data caused by incorrect logic or implementation
- Adhoc data exploration, validation, investigation to help select right tech design and to help Product Management team take right decisions.
- Monitor and troubleshoot production issues with pipelines we own
- Develop and adopt data quality checks to monitor data issues in the systems
- Scope and plan new development: assess level of effort, provide timelines
- Maintain tickets hygiene in Radar (ticketing system)
- Evolve jobs, apps and system we run to a better state in all aspects: code quality, complexity, maintainability, documentation
- Communicate with other data engineers in the team, peer teams (QA, UAT, Platform, etc), with project managers and engineering managers on status, blockers, estimates, timelines, etc. 

**KEY QUALIFICATIONS**
**Must-Have**
- 5+ years of hands-on experience in big-data field: Hadoop (HDFS, YARN or Mesos), Spark
- Excellent knowledge and hands-on experience of SQL in context of Big Data: Spark SQL, Hive QL
- Excellent knowledge of Spark:
- We run on 2.4.x currently. Migration to Spark 3 is upcoming.
- Ability to understand and optimize Spark execution plans via Spark UI
- Excellent knowledge of Scala including Functional Scala Libs (Cats)
- Batch processing and ETL principles in Data Warehouses.
- Data completeness signals and orchestration
- Approaches for historical reprocessing and data correction
- Handling bad data and late data in inputs and outputs
- Schema migrations and datasets evolution
- Strong speaking English. Many times engineers will need to rely on information heard verbally on the meetings. One has to be able to explain own ideas and be understood by native speakers.
- Ability to learn fast new set of tools and technology used internally: Radar, platform services, telemetry providers, Spark-as-a-Service, build system and much more.

**Will be a plus**
- Understanding of functional programming ideas and principles.
- Experience in building and using web services. Part of stuff is talking to internal services to do its job. Sometimes we need to run our own small service.
- Experience with any of Teradata, Vertica, Oracle, Tableau.
- Spark Streaming and Kafka. We have NRT pipelines and their presence is expanding.
- Experience or knowledge of: Apache Iceberg, Trino (Presto), Druid, Cassandra, Blob storage like AWS.
- Experience with Splunk.
- Understanding or experience with Azkaban or Airflow. 

**Worth mentioning**
The hybrid work mode is discussable
Best of the class corporate culture.
 
Apply today for the role to learn more and talk to a project representative!