Good Job Games

DevOps Engineering Case Study

Description
Good Blast is a casual Match 3 game played by millions of users all around the world. The
product team came up with new feature requests to allow users to compete with each other. The
platform engineering team developed a REST API to keep track of users’ data and allow users
to join tournaments and see how they rank against other players and it needs to be deployed on
a cloud platform so that the QA team can perform tests of its integration with client devices. As
the devops engineering team, we are responsible for creating all required infrastructure,
deploying the leaderboard service, and setting up monitoring for the deployed service.
Notes
● Feel free to contact us if you have any questions or need further information.
● If you can not complete within the expected period for a reason, please let us know if
you need more time.
● You may add new features, apply optimizations, consider security and real-life scenarios
and update existing features.
● Submit your effort even if you do not have enough time to complete all functionality.
● Make sure that all created resources are shared as IaC and yaml files.
● Cloud providers offer free credits for new accounts, you may use those for this case
study.
Submission
● Once you are done, submit your work as a zip file including all required files.
● Include a README document that explains your infrastructure design, used
technologies and how to run your scripts.
● Provide a diagram of your system design including names of used resources.

Evaluation
● Deliverables: Your infrastructure provides all functionality described in the
documentation and works as expected.
● Clean Code: Your infrastructure-as-code scripts, CI/CD definitions and deployment
resources are readable, and maintainable.
● Design: Design of your infrastructure, used technologies and applied practices.
● Performance: Response time under high traffic, scalability, and applied performance
optimizations.
Environments
We need separate environments for Testing (test-env) and Production (prod-env).
DELIVERABLE: test-env
DELIVERABLE: prod-env
Infrastructure-as-Code
Before deploying our service, we need to create a new stack to make sure that our deployment
runs smoothly on a cloud platform and is scalable under high network traffic.
● There should be IaC scripts of how our stack is created, so that we can be sure that it
can be replicated when needed.
○ Our networking infrastructure should be described in IaC scripts.
○ Our container orchestration infrastructure should be described in IaC scripts.
○ Our database and container registry should be described in IaC scripts.
○ Our secret store should be described in IaC scripts.
● There should be a role assigned to our cluster that enables access to only resources that
are used by our cluster.
● We should be able to create separate stacks for test-env and prod-env.
● You may use any IaC tool, such as AWS CDK, Terraform, Pulumi, or Google CDM.
● We should be able to destroy/create the stack for an environment easily by running a
simple command.
DELIVERABLE: test-env IaC scripts creating a role, database, secret store, container registry,
kubernetes cluster, domain records
DELIVERABLE: prod-env IaC scripts creating a role, database, secret store, container registry,
kubernetes cluster, domain records
DELIVERABLE: IaC script to destroy a stack
Secrets
Sensitive information such as passwords, access tokens etc. should be stored in a secret store
such as AWS SSM or Vault for each environment.

Build Script & Deployment files
● Create a build script to generate Docker images (Same for all environments).
● Create Kubernetes deployment files for each environment. (Optionally use Knative and
Istio)
● Create required files to expose your deployments to the internet for each environment.
● Create required files for deployments to be reachable via a specific domain for each
environment.
DELIVERABLE: Build script
DELIVERABLE: Kubernetes YAML files for test-env
DELIVERABLE: Kubernetes YAML files for prod-env
Release
A release is a file containing:
● Deployment files for an environment
● Docker image(s)
● Configuration parameters
● Secret store ID
Deployment script
We need a deployment script that takes a release file as a parameter and does the deployment.
DELIVERABLE: Deployment script
CI/CD
We want to deploy our service using a Jenkins based CI/CD pipeline.
● There should be a lint stage that checks code structure.
● There should be a test stage that runs all tests and checks if they all passed and
coverage is above a threshold.
● Runs build scripts to create Docker images.
● Push images to the container registry.
● Create a release file using the deployment files in the current branch, configuration
parameters for current environment and add/commit to repo.
● Runs deployment script supplying the generated release file
● Environments
○ CI/CD pipeline for test-env monitors test branches and is triggered automatically
when a commit is pushed.
○ CI/CD pipeline for prod-env is triggered manually.

DELIVERABLE: CI/CD pipeline for test-env

DELIVERABLE: CI/CD pipeline for prod-env
Security
As the DevOps Engineering team, we are asked to consider the following security concerns and
take action on them.
● Our services should respond to HTTPS connections only.
● Our services should be secure against DDoS attacks by rate limiting requests from a
specific IP. You may assume that the rate limit for a specific IP is 100 requests/second.
DELIVERABLE: HTTPS only for both environments
DELIVERABLE: 200 requests/second rate limiting from a specific IP for dev-env
DELIVERABLE: 100 requests/second rate limiting from a specific IP for prod-env

Monitoring
After we automated the deployment process, we want to make sure that our service is working
smoothly and we are able to monitor the health of our deployed service.
● There should be a monitoring system so that we can trace network events, system
availability, network success rate and some other health metrics.
● There should be an alerting mechanism that notifies our DevOps team on infrastructure
and network failures.
● There should be a logging mechanism that allows our team to trace service logs and
debug service level errors.
● There should be an operational dashboard that provides metrics and logs so that we can
observe performance of our services. It should be deployed as publicly available so that
we can check metrics and logs.
DELIVERABLE: Logging dashboard
DELIVERABLE: Monitoring dashboard
DELIVERABLE: Alerting