Position: Senior Big Data Engineer
Company: Data Science UA
Experience required: 5y
Category: Scala
English level: upper
Published: 2021-09-01T00:00:00+03:00

**KEY RESPONSIBILITIES**
    ✔️ You will help architect and build solutions to business-critical problems
    ✔️ You will be participating in interesting projects such as:
    ✔️ Carrying out efficient integration with our data providers via various API endpoints and data representation formats.
    ✔️ Building and deploying an in-house distributed ETL pipeline for processing petabytes of data per day
    ✔️ Enable an accurate, comprehensive and reliable data storage in our distributed data warehouses based on the needs of other teams
    ✔️ Providing continuous improvements in the way data is being processed and stored based on the feedback and needs of the business or other teams
    ✔️ Setting up monitoring for key performance metrics and overall systems' behaviour to promptly react in case any anomaly detected
    м You will be responsible for optimisation of ETL pipelines, maintaining over 60 Spark jobs. Building a data lake for data scientists and analysts.
    ✔️ Experimenting with new tools and technologies to produce cutting-edge solutions to business problems
    ✔️ Be a part of a self-organising, results-oriented agile team using Kanban to complete new product launches

**You also**:
    💪 have 5+ years experience in software development and have exceptional problem-solving skills
    💪 solid hands on Scala programming language (proven production experience)
    💪 solid hands on Apache Spark framework (proven production experience)
    💪 solid hands on Apache Spark SQL
    💪 have proven production experience with at least on of the following big data warehousing engines (distributed storages): Apache Impala and/or Presto and/or Amazon Athena; queries optimisation, data analytics
    💪 have proven working record with RDMS (relational databases) (e.g. PostgreSQL/MySQL (stored functions, triggers, command line tools))
    💪 have proven production experience with at least one of the following resource sharing frameworks Apache Mesos and/or Kubernetes and/or Yarn: jobs deployment, jobs scheduling, resource allocation, jobs execution debugging, logs collection and analysis
    💪 have proven production experience with monitoring tools and distributed logging frameworks (Prometheus, Grafana, GrayLog)
    💪 have proven production experience with at least one of the cloud services providers AWS Cloud services and/or Google Cloud services: command line tooling , data access, integrations, authentication etc.
    💪 are Unix OS family (Linux/FreeBSD/OSX) advocate: shell scripting and essential  commands knowledge
    💪 are familiar with Docker and use it in your integration tests
    💪 are passionate about following best industry practices: TDD, BDD , CI/CD etc
    💪 have understanding and working production experience with HTTP/HTTPS protocols; ability to setup and deploy a simple web server if needed; 3d party HTTP/HTTPS integrations;
    💪 have proven experience of building production-ready ETL processing pipelines
    💪 have a proven track record of building integration with third parties via various APIs (http, file, message base)
    💪 are passionate about working in a fast paced, startup like environment.
    💪 are a proactive self-starter who works well independently and as part of a team
    💪 have good written and verbal English communication skills

**You earn bonus points for the following**:
    ✨ experience with striming frameworks Apache Kafka and/or Apache Flink
    ✨ experience with distributed key value stores: Aerospike, HBase etc
    ✨ experience with Python and JavaScript
    ✨ experience with workflow frameworks (e.g. Apache Airflow etc.)
    ✨ some experience in the AdTech sector is advantageous
    ✨ previous experience in Bank or any Accounting field in a role of analyst is an advantage
    ✨ experience with working in an international team

**What is in it for you**:
    🏆 Flexible working – We give you the option to Work from Home or better still you can work from any  office for 2 weeks in total per year.
    🏆• Sharing culture - If you have learned something new, we welcome you sharing to the to share it with the team through a short presentation.
    🏆 Diversity – You will be working in global diverse teams with intelligent and like-minded individuals
    🏆 Days off - You will get 20 days holiday per year plus 2 bonus days off including your birthday. We get all 11 of the Ukraine public holidays off 
    🏆 Limited Company - We help manage your Limited Company and provide you with supporting where needed