Position: Lead Big Data Engineer
Company: EPAM
Category: Data Engineer
Experience required: 3y
English level: upper
Published: 2023-03-01T00:00:00+02:00

The main purpose of the project is to improve performance and reduce operational costs for the Connection Factory application through new architecture and clarified application lifecycle management.

Job Description
1.    Building from scratch data ingestion and data integration solution
2.    Building data processing pipelines
3.    Building solution for monitoring of data processing
4.    Building data quality solution
5.    Application engineering to integrate big-data processing solution with FTP data source using web application as a mediator
6.    By default Data Engineering solution will be cloud agnostic
7.    Possible integration with several public cloud providers starting from Google Cloud Platform
8.    Possible integration of data processing solution with existing data platform and / or several production use cases
9.    Implementing unit tests, when necessary
10. Containerization of data processing solution
11. Implementing CICD pipelines

Skills requirements:
1. Understanding of Data Architectures and production experience in implementation:
2. Experience in implementation Data Ingestion / Integration solutions, including reusable CDC solutions
3. Data Engineering
4. Technology
a. Apache Spark, PySpark could be an option
b. ElasticSearch + Logstash
c. Prometheus
d. Experience with Grafana tool
e. Apache Kafka
f. Experience with Airflow or Celery would be a big bonus
g. Application Engineering
i. Java (scala) background with Monolyth / Microservice / SOA development experience
h. Understanding of basic principles of containerization and container orchestration. Docker / Kubernetes / Helm / Service Mesh / K8s Operator experience is nice to have.