Discussion: What practices should we adopt to help train new on-callers?

**Goal:**

This is a pitch for adopting a structured learning path for new GitLab team members (SRE or non-SRE) to incrementally accumulate the knowledge necessary to successfully fielding most on-call issues.

In this context, "structured learning path" means gradually building a working knowledge of the system and how it is maintained, beginning with immediately applicable knowledge and iteratively expanding in breadth and depth, with specific learning objectives at each stage.  Students should be able to use what they've learned to answer questions about how components interact or next steps to diagnose scenario X or how to get help with investigating or fixing component Y.

Lots of folks in our organization have experience being on-call (either here or elsewhere), witnessing what worked well for themselves and others.  Personally I feel strongly that catering to several different learning styles is essential to being kind and inclusive to our current and future teammates, so I'd love for folks to contribute their ideas, experiences, and opinions on what practices we could adopt that would be most helpful for supporting new on-callers (and ideally also promote continuous learning that benefits tenured on-callers too).

**Starter material:**

There are several ways we could organize both existing and new training materials.  Whatever method we choose, there should be a clear progression from introductory to advanced topics, each of which should have explicitly stated learning outcomes that tie into the overarching theme of supporting on-call duties.

Some examples:

*Intro level:*

* Learning objective: List all major components of GitLab.com's service stack, and briefly describe the role of each component.
* Learning objective: Know basic usage of these 5 observability tools.  Understand what kinds of questions each can answer.  Describe a scenario in which this tool would provide useful information.
* Learning objective: Know how to find help for troubleshooting a problem with a component or scenario you are not familiar with.  Where can you find a list of subject matter experts?

*Intermediate level:*

* Learning objective: Know how to use the rails console, specifically with the GitLab object models.
* Learning objective: Walk through 3 of the curated historical postmortems to gain familiarity with common patterns of troubleshooting the GitLab.com stack.  Include at least 1 application regression and 1 infrastructure failure.  Become acquainted with the tools and tactics used for diagnosing and fixing these regressions, as well as the styles of collaboration and engagement with peers during the troubleshooting and remediation phases.
* Learning objective: What does "normal" looks like at a system- and component-level?  View this from each of the vantage points you would use during troubleshooting: Grafana dashboards, host-level command-line tools (perf, netstat, etc.), rails console, psql, redis-cli, etc.

*Advanced level:*

* Learning objective: What does "abnormal" look like?  Limit the scope of this to just your chosen areas of focus.  Know what are the most common or most critical known failure modes for that service or component.  What are the symptoms, side-effects, and remedies are for those failure scenarios?
* Learning objective: Which components of the architecture are currently the most fragile?  Which are singletons?  Which services lack graceful failover?  How does this affect our SLO, and is our recovery time reliably low?  Where can you quickly find the documented recovery procedures for these critical components?
* Learning objective: Know how to scale out a service tier with Terraform.  Know how to infer from effects whether this was helpful or harmful, and if harmful how to identify the critical downstream bottleneck.

Story points: 3.0