Position: Senior DevOps (Kafka)
Company: Alcor
Experience required: 3y

Responsibilities

Take ownership of Kafka clusters, review the current implementation, and work to implement and automate best practices across all teams and further improve observability and monitoring of the clusters.

Evaluate alternatives to Databricks for running spark jobs and notebooks.  Ideally, the alternative environment should run jobs on top of Kubernetes and should include logging, monitoring, and observability.

Review People.ai’s Hive Metastore and data lake implementation and evangelize best practices and work to improve the performance of the environment.

Support all teams in the adoption of Kafka and streaming stack as the company makes a big push from batch processing to streaming.

Standardize implementation of RDS Postgres through best practices, and improving availability, reliability, and visibility of the environment.

Requirements

Demonstrated focus on automation

Proven track record of partnering with different engineering teams and completing company-wide initiatives in areas such as automation, security, migrations, and monitoring.

Deep systems engineering and computer architecture background (e.g. Operating Systems and Networking) 

Big data stack such as Kafka, Hive, Data warehouses, Hadoop (Cloudera, EMR) deployed in AWS, ElasticSearch.

Use of Terraform and Ansible to automate resources in the public cloud.

Implementation of monitoring and observability in the stack.

Nice to have
 
Experience in managing and performance tuning of Postgres instances.