Position: Integrated Operations Center Shift Engineer/Application support
Company: Zoolatech
Category: Sysadmin
Experience required: 1y
English level: upper

Our client is a Danish jewelry brand, and one of the most famous jewelry brands in the world. We are building a team to help improve the systems’ reliability and develop great partnerships for years. 

Pandora is looking to build an Integrated Operation Center to improve its customer focus and business outcomes.  The IOC will initially provide incident management and production support. This effort is part of Pandora’s DevOps journey in which 24/7 monitoring of its platform and support for engineering is key. Pandora’s goals under its DevOps journey include improving productivity, creating efficiencies/savings and generally strengthening its culture. The IOC will bring a culture of accountability, reliable follow up and strong SLA compliance. And it will be core to moving towards that ‘single pane of glass’ for the performance of services across Pandora. 

 Similarly, the IOC will improve and implement Application Performance Monitoring (APM) at Pandora leveraging NewRelic. The goal is to achieve greater operational performance, help the engineering team to achieve greater focus and drive a better customer experience. Together the teams will improve key business metrics and accomplish Pandora’s ultimate goal of a leading customer experience and stronger business outcomes.

The core IOC operations will be based in Mexico (1x8h shift) and Turkey (2x8h shifts) in follow-the-Sun setup.

Responsibilities


Work in shifts to cover 24/7 support of customer's applications

Able to work in rotating 8 hour-long Morning/Evening shift

Monitors Alert management tools and acknowledges P1/P2 alerts

Performs initial alert triaging, runs runbook steps, goes through checklists, runs initial diagnostic scripts to detect false positive and prevent unnecessary escalations

Performs initial escalation to SRE/Senior IOC engineer and when true positive Incident is confirmed - declares incident, updates status page, starts collection of post mortems/RCA log, escalates to development teams, gather war room and implements workaround/performs rollback when necessary

Reports to IOC Team Lead/SRE Manager

Takes part in regular trainings, on-boarding and KT sessions

Updates outdated/missing Operations documentation, SOPs, runbooks,  checklists

Updates Alerts, Thresholds, tags, aggregation/deduplication rules to decrease noise of Alert Management systems

Tracks planned maintenance that coincides with his/her shift

Logs Incidents/Alerts he worked on during his/her shift

Re-assigns/closes Incident tickets/mutes acked alerts during shift handover

Submits individual shift report, and Shift Handover report at the end of the shift, syncs with engineer on the next shift

Constantly monitors and improves team’s KPIs - like decrease MTTD/MTTA and increase number of Critical Alerts/Incidents responded per person per shift, decrease # of escalation, increase # of Incidents resolved on first hit

Self-learning & extreme ownership attitude

Skills Required

Graduate or last year University student of Computer Science or similar faculties (cyber security, IoT, Computer communications)

Good understanding of ISO/OSI model, encapsulation, Network protocols used by web, dns, tcp/udp, HTTP/S, REST API applications

Experience in using monitoring/observability solutions like NewRelic, DataDog, Splunk, AppDynamics, Instana

Experience in using on-call rotation and Alerting management platforms like OpsGenie/ PageDuty

Incident management skills and troubleshooting production web-/enterprise application issues

Understands Incident/Problem resolution flow, escalation criteria and key NOC KPIs

Experience in writing team documentation and efficient runbooks/SOPs for production web-applications/API- and cloud services

Fluent with Linux/Windows CLI tools used for network apps/db troubleshooting: ping, traceroute, curl, nc, ss, dig, nslookup, tail, grep, nmap, tcpdump, telnet, sql client, postman etc

basic scripting with bash/shell script/powershell and Ansible or similar tool

SQL fundamentals

HTTP/S methods and Error codes

Chrome Developer tools