Discussion and high-level plan for Chef to Ansible transition

With Chef Server now EOL, limited options for community alternatives, and an overall pessimistic outlook on the future of the product/community after the sale of Chef Software, we have decided to begin the transition to Ansible next quarter. This issue will be used to kickstart the discussion around gathering some initial requirements, and generate the high-level plan for the effort that will be fed back into &392.

## Basic plan

Here is a roll-up of the (very) high-level plan with additional/more detailed notes below. This is still a work-in-progress, and will likely continue to evolve even as we proceed to implementation and begin iterating on rolling through specific portions of the infrastructure.

1. Reorganize gitlab-com/gitlab-com-infrastructure> and update README
1. Extend pipeline configs in gitlab-com/gitlab-com-infrastructure> to support ansible / [GET](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit)
1. Implement [tagging standards](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11312) with additions as needed for this project
1. Leverage tags/filters for [GCP inventory](https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_inventory.html)
    1. This should not necessarily be the same for [deployment](https://gitlab.com/gitlab-com/gl-infra/deploy-tooling), consul/service discovery may be a better option to distinguish between new, unconfigured nodes and those which have been successfully bootstrapped
1. Develop bootstrap process, plan/implement corresponding terraform changes to support both Chef & Ansible
1. Design equivalent top-level playbook structure for existing Chef roles
1. Begin migrating common/base configuration and roles
1. Design, document, and implement GKMS/GSM equivalents to Chef Vault items
1. Rollout sequence (WIP)
    1. Leave roles that are/will be migrating to k8s for last
    1. Bastions
    1. Gitaly
    1. Praefect
    1. Patroni
    1. Redis
    1. Console
    1. Consul
1. Develop plays for current configuration in chef cookbooks, by role
1. Cleanup unused Chef "stuff"
    1. Roles
    1. Cookbooks
    1. Data bags
    1. Archive projects in gitlab
    1. Decommission chef.gitlab.net
    1. Remove monitoring/backups
    1. Audit/cleanup documentation

## Notes
### Inventory

Early on, I had considered using Consul as an inventory source, but later realized that we would encounter a circular dependency on bootstrap, where nodes not already registered in Consul would not be visible, and as such Ansible could not target them to run a playbook that would... register... them... in... consul. While expanding our use of consul service discovery may be useful in other areas, it is not appropriate for this case. Using consul as a pivot point for a successfully bootstrapped node _may_ make that a more appropriate inventory source for deployments, however.

Which brings us to the [Google Cloud Compute Engine inventory source](https://docs.ansible.com/ansible/latest/collections/google/cloud/gcp_compute_inventory.html). This plugin will allow us to dynamically build inventories based on filters for matching resources within the GCP environment. We currently rely on Chef as our inventory source for deployments, so both the deployment tooling and the new Ansible infrastructure will need a new source of truth for their inventories. It clearly makes sense to shift to a GCP filter/tag-based approach for bootstrapping, as those values are assigned by terraform and provide a clean method for handoff between terraform and ansible.  We already [have plans](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11312) for implementing formal tagging standards, which should also be leveraged here, as well.

For deployments, however, leveraging Consul as the next "higher-level" inventory source seems appropriate, as a node would be registered in the service catalog and be subject to health/liveness/readiness checks indicating that a deployment to that specific node is appropriate, whereas a newly bootstrapped node may not have all pieces fully configured. Subsequent ansible runs could bridge the gap between the two should a new node be in the process of being bootstrapped _during_ a deployment, or we can incorporate functionality to prevent such situations into our deployment workflow.

Implementing the tagging standards to facilitate updating/instantiating the inventory using a new source of truth (GCP) should be one of the first things to be done. The current Chef inventory can be kept for the duration of the project (ie minimal bootstrap, empty run-lists), but ultimately the new inventory will need to be well understood and fully integrated by the end of this project, with an ability to easily disable and remove chef once we have everything shifted to ansible. Similarly, developing plays to register services in consul and develop corresponding health checks should follow immediately after, so that we have that framework in place to adopt as we shift from Chef and Chef inventory to Ansible.

### Pull vs push model

From an architecture standpoint, one of the most fundamental differences between Chef and Ansible is the communications paradigm. While Ansible [can support a pull model](https://docs.ansible.com/ansible/latest/cli/ansible-pull.html) and we should certainly consider that if we encounter scaling challenges, the default starting assumption for most of the documentation, examples, tutorials, and small-scale examples rely on a push model via SSH. This is how our deployments to gitlab.com are conducted, and for simple alignment with the overwhelming mass of documentation and examples, is probably where we should start.

In addition, while not intrinsically related to this effort, we would benefit in several areas from leveraging Ansible in a build pipeline for "golden" images used in dynamically scaled infrastructure; we should keep that in mind as we work through this project. Because of its agentless nature, Ansible lends itself well to this approach, and that is a likely next project after the initial switch.

### Management and orchestration

As I started reading about Ansible, one of the first questions I had was whether we need/want to leverage AWX (or Tower). After skimming through the main product page, I don't see a very compelling need for this in our case, especially based on the success we already have had in using GitLab for most of the primary features listed at https://www.ansible.com/products/tower for [deployments](https://gitlab.com/gitlab-com/gl-infra/deploy-tooling) and more recently for [codifying/automating database operations tasks](https://gitlab.com/gitlab-com/gl-infra/db-ops).

| Feature | GitLab equivalent |
| --- | --- |
| Ansible Dashboard | Metrics exported to Prometheus and Grafana dashboards |
| Real-time job status updates | GitLab CI Jobs |
| Ansible Tower Workflows | GitLab CI Pipelines |
| Activity streams (auditing) | GitLab CI Pipelines / repository history |
| Ansible Tower clusters | N/A |
| Integrations | GitLab CI Pipelines with webhooks/integrations |
| Ansible Tower Smart Inventories | N/A - at most, <a href="#inventory">dynamic inventory</a> will suffice for our needs |
| Ad-hoc jobs | GitLab CI Jobs |
| Remote command execution | GitLab CI Jobs (or Teleport) |
| REST API | GitLab API |

### Secrets

With the recent completion of initial setup for Google Secrets Manager, we can begin coordinating the shift to begin using that service. I have not dug into this in sufficient detail, yet, but [some initial work](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12275#note_480745661) to update the instance service accounts will need to coordinated with this effort, as well as with the OS upgrades. In addition to updating instance service accounts, we need to functionally replace all existing tooling for managing secrets in GKMS and applying via Chef Vault. I have not really explored this in-depth, yet, so this is definitely a case of "here there be dragons" due to being so inherently vague.

### Project structure

As we have seen over time, breaking out similar codebases and modular assets into multiple repositories enables flexibility when you have many consumers of the modular code, but this inherently brings a certain amount of added complexity and management pain. I have two thoughts to address this for this project.

First is the recently announced [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit), developed by our QA team to effectively provision GitLab environments on-demand for testing. We need to evaluate the included Ansible code from that project due to the clear, overwhelming overlap between the two efforts. Absent any insurmountable complications, we should try to avoid forking to make the most effective use of a shared resource, but could also consider forking and maintaining a descendent project if absolutely necessary. Alternatively, we could attempt to cherry-pick the portions of Ansible code that apply to both situations, and work on a separate project from which both GET and our infrastructure tooling could inherit. In either case, I am hopeful that a closer examination shows this as an effective means of sharing infrastructure provisioning code both within GitLab and with the broader community.

Second, and possibly independent of the first, I'm thinking that our Ansible tooling need not be completely separate from our current Terraform code, and that consolidating both projects under gitlab-com-infrastructure could bring multiple benefits. It would provide the reduced complexity and ease of management of a monorepo, easier coordination of changes between Ansible and Terraform code for bootstrapping instances / managing instance groups, and provide a single place where most core infrastructure code and tooling can be stored/managed/documented. Consolidation of our Terraform code back into a monorepo with strict(er) adherence to feature-flagging code changes would necessarily be included in this approach to get the most benefit. Improved consistency across and within our infrastructure code is certainly desirable, but is not implicit from either approach. In the best case, the consolidation would make it easier to find portions of our codebase and therefore make it easier to keep things consistent (this is definitely a reach, however). This idea is not _completely_ fleshed out, and there are implications with things like our ephemeral and gitlab-services environments, but those too could conceivably be consolidated for the same reasons (though with a fair bit more difficulty, due to differing deployment patterns). Additionally, this centralizes a significant amount of privilege and therefore security risk into a single project, so maintaining good code hygiene and proper access controls would be critical.

While I am in favor of the second point, it is the most significant decision I have yet to (fully) make, is not fully fleshed out (as noted), and would benefit the most from feedback from the broader team. If we decide that a shared infrastructure repo is not the right approach, I am still strongly in favor of keeping our Ansible code and tooling in a monorepo, regardless. If we have a clear need to distribute code in consumable format outside the team and/or outside of GitLab, we can revisit this choice later, but for our own team's immediate usage, I think the benefits far outweigh any drawbacks.

### Disaster Recovery

When I first approached conceptualizing this, I was thinking far more within the centralized model of Chef, and leaning towards implementation of AWX/Tower. Once I worked through the table above, it was easy to see that we shouldn't need that model. With that in mind, we will need to structure this project and locate the corresponding CI pipelines/jobs in much the same way that we do for Chef or Terraform today (e.g. on the ops instance, with a mirror to .com). Thus we would need to look to the DR capabilities from the underlying GitLab instance hosting the project and its CI pipelines.

### Other Outstanding Questions/Topics

There are a few aspects that I can still think of, which I haven't touched on too much, yet.

#### "Service-ness"

So far, this is very squarely in the realm of implementing a tool, and a corresponding technology stack. As there is no central server, and it doesn't really have a well-defined "interface" for interacting with the tool beyond directly accessing the code and automations triggered/run via CI, I hesitate to push this through a standard production readiness review, but at the same time we should reference our template to ensure that we check all the boxes for the relevant bits that still apply (documentation, architecture diagrams, monitoring/logging/alerting, etc).

#### Observability

This is an area where we have interesting challenges, and not much to compare against in our current system. With Ansible being an agent less system, much of the monitoring will likely need to be baked into the CI jobs, and handled via hooks to external services at runtime. There is no central server to collect and emit logs to a traditional logging infrastructure, and likewise the same with metrics.

Story points: 8.0