nyuuzyou/wikis huggingface dataset:
Wikimedia Cloud Services team/Alerts

This page gives some information about WMCS team alerting system and procedures. All the team alerts go to this same alerting system.
= Current alerts =
You can see the list of current alerts being triggered in two places:
* Wikimedia wide alertmanager, filtering by the 'team=wmcs' label.
* VPS alertmanager, filtering by the projects you might be interested in
= Old alerts =
You can check the old alerts in this logstash dashboard.
= Ideal scenario =
Ideally every alert will have two links, one called 'runbook' to the detailed steps on how to handle that specific alert, and how to troubleshoot and fix the issue the alert was triggered by. More info in Portal:Cloud_VPS/Admin/Runbooks
And another link called 'dashboard' linking to a relevant grafana dashboard.
= Current status (when writing this) =
Currently we are transitioning to a prometheus + alertmanager only setup, but we still have some alerts in the old icinga instance. So there's still some alerts that either don't have a runbook
= Handling a page =
== If you are close to a laptop/ device with a browser ==
When you get paged, the first thing to do is to go to the Wikimedia alertmanager dashboard.
From there you can see all the alerts that are triggering, there you can ack the one/ones that are paging by clicking on the tick mark on the alert:
Or creating a silence with a comment that starts with "ACK! " and adding there some information, like the task number you are using to track the incident.
== If you are not close to a laptop ==
Then, if you are sure you will be able to handle it in a short time, you can ack the page on the splunk mobile app or via sms.
Otherwise, you can just let the paging system wake up the next person in the rotation to handle the incident.
NOTE: If you acked the alert on splunk, remember to also resolve the incident on splunk, otherwise it will page again in 24h if the alert is still there.
== Special temporary case: if the alert has the label 'source=icinga' ==
There's still some alerts that page through icinga, if the alert that is paging you has that 'source=icinga' label, then you will have to go to icinga to silence it, to do so:
*click on the elapsed time collapsable (to the lower left in the alert box)
*click on the 'wikimedia.org' link, that will open the icinga page for that alert
*in icinga, click 'Schedule downtime for this service'
= Related Links =
* Wikimedia wide alertmanager
* VPS alertmanager
* WMCS grafana dashboards
* Portal:Cloud_VPS/Admin/Runbooks
* Portal:Cloud_VPS/Infrastructure
* Portal:Cloud_VPS/Admin
* Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T310598_Team_oncall_alerting_schedules_and_processes
Wikimedia Cloud Services team/Coding

We write a lot of code, so it seems like that would be the place to introduce some common practices for the WMCS team. In the interest of clean, readable code across the team, it is common practice by development groups to share formatting and other development practices in order to facilitate collaboration. A common example is shared linter settings, like what is used in operations/puppet for puppet-lint.
This is intended as Team Practices, not an overall standard, variations and other settings might exist in our codebases and that's ok.
We usually prefer keeping the defaults of the tools we use relying on specific configurations only for special cases.
For an up ta date set of configurations/tools, see the example repos (and for easy copy-pasting!), here's some common tools used at the time of writing this are:
=== All ===
Example repo: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api
* pre-commit to run linters/static checkers/code formatters (see example code repo for the list).
=== Shell scripts ===
Example repo: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api
* pre-commit shellcheck
=== Python ===
Example repo: https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api
* tox to run pre-commit + tests
* pre-commit mypy check
* black+isort+flake8 / ruff
=== Golang ===
Example repo: https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api
* blubber test image variation runing go test
* pre-commit golangci-lint check
=== Other guidance ===
The movement has mw:Manual:Coding conventions/Python which provides some additional guidance for python specific coding.
Category:Cloud Services
Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/2020-10-07-checkin

https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh
Agenda:
*Q2 Shared Planning
Action Items from last time:
*Determine how to prioritize security concerns of both SRE and WMCS sooner than currently proposed. This includes closing the dmz_cidr loophole this fiscal year per OKR's for SRE.
What services connect from prod to cloud?
Three use cases:
 * Rate Limiting
 * Data replication from prod to cloud
 * Access Control
 
Ideas:
 * Seperate data and control planes for NFS servers -- no L3 connection for data between cloud and prod
 * Move L3 IP addresses for NFS servers inside of cloudgw umbrella (NAT exception remains, but won't cross to prod)
 * Migrate NFS data to ceph cluster and utilize cinder volumes (lack storage today for everything, especially dumps)
 * Create a specific cloud VLAN
 * Move all NFS servers to public IP's (all but 1 (+1 shadow) is already public)
Q2 Planning
 * Arzhel / SRE will have limited time in Q2 due to resourcing and holiday
 * time for PoC?
 ** best-effort only
 * shift gears and focus on the separation ideas above; less interconnected work that won't block on Arzhel
PoC
* close to being finished from switch config point of view. Only missing piece is seeing traffic flowing
** beyond that WMCS won't require Arzhel
Feel free to use other SRE Resources as needed
Questions:
*why the focus on NFS?
**We don't get any benefit until the entire proposal is finished. No incremental benefits to close smaller services
*How to manage a machine that crosses the realms (prod, cloud)? Isn't there still escalation paths if they sit in an in-between space?
**Yes, at some point need puppetmasters and monitoring. Data replication is tricky. At some point that data has to traverse boundary (prod->cloud) in order to provide data services. In particular mysql replication requires authentication (wikireplicas).
*Does IPv6 mitigate concerns?
**Right now cloud servers access private IPs. Even with ipv6, this access would continue
*Hybrid solutions perhaps less desirable?
**Separating data and control would allow for boundary designs
*Do one or more of these ideas help provide work on SRE OKR?
**Yes, seperate the giant ACL exception is the goal. WMCs should pursue ideas in this space and plan them for this year.
Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/2020-11-10-checkin

= 2020-11-10 WMCS network checkin =
* Status updates
* Questions, feedback?
* Next to do's
== status updates from arturo ==
* cloudsw in codfw could be interesting
** test changes before introducing in eqiad
* cloudgw PoC in codfw going well:
** neutron accepted to work without doing the SNAT, now being done by cloudgw.
** VMs now use floating IPs (if they have one) for connections outside the cloud. No shortcomings detected so far, other than refreshing firewalling and other ACLs.
** neutron is happily running without our custom hacks.
* next steps, address some limitations:
** no NIC bonding in data plane. Would like to try it out, requires DCops work for the additional cable patch.
** no HA. Would like to try it out, requires another server in codfw.
* next next step: start thinking on eqiad1
** if we are fully happy with the codfw setup, then we should think on eqiad1
** this requires procuring 2 HW servers (could be misc spares with 1x1GB NIC 2x10GB NICs)
** our plans for NFS will likely leverage this setup: https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/notes/NAT_loophole/NFS
== Questions ==
* faidon: netns doesn't provide enough isolation of both realms (context: NFS design ideas)
** arturo: openstack does exactly the same in other services
** brooke: NFS doesn't provide direct escalation paths
** arturo: we are asuming risks when running the cloud in the same DC as prod, we share stuff. Openstack bridges things in nature.
* faidon: defense in deph approach
* andrew: we take prod management for granted. Is this in scope?
* faidon: can we drop VMs reaching the wikis using cloud addresses now?
* arturo: not in our KRs for this quarter
* arzhel: can we please work on reviewing ACLs this quarter? https://phabricator.wikimedia.org/T264993
* arturo: yes, I can do that
* birgit: key takeaway for NFS, how many layers of security we are adding vs the current model
* brooke: we are in the early stage of NFS evaluation: we are in the research phase, not in feedback collection phase.
* birgit: can we have a 3 top goals for the NFS project? like guiding principals, to make sure we keep on track (i.e.:high performance for toolforge users, more security layers etc).
* faidon: add realm boundaries to diagrams, to make sure where prod/cloud start/end.
* arturo: any concerns with the cloudgw project moving forward?
Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/2021-01-12-checkin

= 2021-01-12 WMCS network checkin =
== agenda ==
* Q3 goals sync
** make sure we know each other Q3 OKRs and such
* procurement of hardware for new edge network setup:
** codfw cloudgw device procurement https://phabricator.wikimedia.org/T268016
** eqiad cloudgw devices procurement https://phabricator.wikimedia.org/T270705
** renaming labtestvirt2003 to cloudgw https://phabricator.wikimedia.org/T271519
** codfw cloudsw TBD
* Production Cloud services relationship
** https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Production_Cloud_services_relationship
== notes ==
Q3 OKRS:
 Cloud Services:
 1. Finish introducing cloudgw into the network (see HW phabricator tickets)
 2. Remove two NAT exceptions
 3. Establish guidelines for cloud and production relationship (deliver as we go, publish scenarios as we agree)
 SRE
 1. New resources (network engineer!)
 2. 10G / DC Upgrades
Faidon:
 In general, let's try and limit the number of open items / task under consideration. Focus and finish things to completion.
 Front-line defenses continues to be important, especially in todays climate
 Still trying to catch up from Q2, time continues to be limited
 Can we delay cloudgw? What would this block or prevent?
Brooke:
 The conversation will change architecture decisions. How to handle NFS, wikireplicas, and other needs going forward. However, cloudgw is mostly an implementation piece.
Nicholas:
 The design / understanding for the relationship is important and shouldn't be delayed.
Arturo: 
 People don't know how to cross realms. So ideas are not pursued because they feel it's impossible. Proposal is the provide guidelines for how to introduce services that have cross-realm data needs. And it can be an evolving document.
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Production_Cloud_services_relationship
Faidon:
 Need similar guidelines for production. Some should use anycast, some should use LVS,etc. This document could link to a similar abstraction to production guidelines.
 Cross-tenant? Cross-region? Is this layer2 or layer3 (tunnel or isolated?)
 No answers today, but multi-dc is coming.
 In VM and in cloud. Not in VM and not in cloud. Anything in-between or mix?
Arturo:
 Openstack Ironic does this. Proposal is poor man's implementation of this.
Faidon:
 We are likley to see more of this. There's always been a gap to "test" a new thing. You can do VM's or be in production. No real testbed.
 Where does toolforge fit?
Arturo:
 No plans to run toolforge outside of cloud vps. Distant feature is ??
Brooke:
 Toolforge is interesting because it touches every case. Virtaulized on VM's. Then has NFS partly utilized to get credentials for production DB's. Also talks to wikireplicas in production. No plans to move out. K8's network overlaid on virtual network.
 
Faidon:
 Any additional overlay?
 
Brooke:
 Yes, service network on top using calico.
Wikimedia Cloud Services team/EnhancementProposals/2020 Network refresh/2021-02-24-checkin

= 2021-02-24 WMCS network checkin =
== agenda ==
* https://phabricator.wikimedia.org/T271058
** buy new boxes? WMCS to decide 
* https://phabricator.wikimedia.org/T209011
* procurement of hardware for new edge network setup:
** codfw cloudsw https://phabricator.wikimedia.org/T272348 (needs discussion)
* Cross-Realm traffic guidelines document (previously known as Production Cloud services relationship)
** https://docs.google.com/document/d/1aCKi6zsDGftUWeCIj79B0XHWYAbbyKT1HuWHQleLV2E/edit?usp=sharing
** https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Production_Cloud_services_relationship
== notes ==
* Faidon: on cloudnet1004 HW issues: https://phabricator.wikimedia.org/T271058 feel free to request new boxes in case we need to
* No more exceptions for dumps. Two exceptions dropped! ;-)
* CloudNAT https://phabricator.wikimedia.org/T209011
** Faidon / SRE ready to help
** WMCS will look again and reach out
* procurement of hardware for new edge network setup:
** codfw cloudsw https://phabricator.wikimedia.org/T272348 (needs discussion)
** No existing underrun for this
** Concern about time and what the need is for? (testing? to match prod?)
** Ok to be Q4 or later
** Purpose: Dallas is mirrored setup of production deployment in eqiad. Used as testbed for making changes. Want to ensure Dallas matches the changes in eqiad with cloudgw boxes
** This would not be be a part of any "second" region for cloud.
** This sounds potentially expensive. Can we reduce the need for testing replica? Can it be virtualized?
** scale is different, it's a minimal representation
** For SRE, "testing" in production, use incremental changes. No real testbed exists. Encourage to find ways to do this.
** Second region could remove need for this
* Cross-Realm traffic guidelines
** Proposal to publish the 3 common use cases, with the last complex case as a DRAFT
** Request to create a list of services, and current thoughts on how they might map
*** Want to avoid forcing decisions before we understand them
** Would be useful to understand the scope of the problems and services, suggesting a wikitable
** End goal is to be able to craft new services utilizing the new document
** Hard to map services to cases, but could map use cases to cases
** Last case is like a loose HV, manaully managed with puppet, not controlled by neutron
** "we can map flows more than we can map services"
** Toolforge is a composition from many services, like NFS
Wikimedia Cloud Services team/EnhancementProposals/Decision record T325382 Choose a lang for the toolforge build service API

NOTE: Follows the discussion to move to an API design or not here phab:T326136
Origin task: phab:T325382
Date of the decision: 2023-01-17
People in the decision meeting (alphabetical order):
* User:Andrew_Bogott
* User:Arturo_Borrero_Gonzalez
* User:David_Caro
* User:FNegri
* User:Raymond_Ndibe
* User:Slavina_Stefanova
== Decision taken ==
Option 1.
=== Rationale ===
We decided to proceed with Option 1, without consensus but majority of the votes.
== Problem ==
We have decided to create an API for the toolforge build service, we need to choose a language for it.
=== Constraints and risks ===
* We have considerable python knowledge in the team
* We have little golang knowledge in the team, this might increase the time to build it
* The k8s ecosystem is built around golang, this might ease the maintenance if golang is chosen
* Some of our components are written in golang (all the admission/validation hooks), this might help maintaining those if golang is chosen
* Toolforge Job service is written in python, and has a similar design, this might help reusing and or sharing knowledge if python is chosen
* There's no strong investment either way for toolforge components, though there's no trials of API design with golang, this could give some experience on both languages before (if) deciding to converge all components to one lang/tooling
NOTE: Full list of libraries here phab:T325382#8488267
== Options ==
=== Option 1 ===
Implementing it in golang.
Pros:
* Golang k8s libraries are the only ones manually developed by upstream, and the main upstream libraries that they design for, they have extensive concurrency support, they are typed, and have testing helpers, all the others are ether automatically generated or not maintained by upstream.
* We gain experience with golang component design
* We gain experience with golang as a whole (and better support current golang components)
Cons:
* Some initial investment is needed to develop the API
=== Option 2 ===
Implementing it in python.
Pros:
* Python knowledge in the team makes it simple to use best practices
* We can reuse some of the code already existing for toolforge-jobs-framework
Cons:
* The python k8s libs are autogenerated from the golang ones, don't have builtin async support (they have a limited concurrency API, not using the builtin python one), type hints, or testing helpers.
Wikimedia Cloud Services team/EnhancementProposals/Decision record T326136 Toolforge build service to move to an API design

Origin task: phab:T326136
Date of the decision: 2023-02-06
No decision meeting needed, people commenting in the task:
* User:Arturo_Borrero_Gonzalez
* User:David_Caro
* User:FNegri
* User:Nskaggs
* Slavina Stefanova User:Slst2020
== Decision taken ==
Option 2. Moving to an http API design
== Problem ==
The current design relying on the cli to interact with k8s makes it hard to implement certain very desirable features like:
* Users to see build logs
* Quotas for builds
* Multiple interfaces (Potential UI + cli)
* Local cli execution (outside of the bastions)
Note that the scope of this decision is only about moving to an API design, the details of which language, etc. will be discussed later on in phab:T325382 if we decide to go for it.
=== Constraints and risks ===
* Developing certain features (starting with the above ones) will become harder and harder
* Duplication of code implementing business processes will become a must for multiple interfaces (ex. cli, web, ...)
== Decision record ==
Option 2 was chosen, move to an API design.
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T326136_Toolforge_build_service_to_move_to_an_API_design
== Options ==
=== Option 1 ===
Leave it like it is now.
Pros:
* No immediate effort needed
Cons:
* Increased and potential blockers implementing future features (see the list in the description)
* Increased maintenance for more than one client platform
=== Option 2 ===
Create an API living in k8s that will take care of interacting with the k8s service, and expose all the needed functionality to the clients through an http API.
Pros:
* Unblocks many features
* Eases the development and maintenance of different clients
Cons:
* Some initial investment is needed to develop the API
NOTE: This discussion started at phab:T325382
Wikimedia Cloud Services team/EnhancementProposals/Decision record T335979 Toolforge envvars/secrets service

Origin task: phab:T335979
Date of the decision: 2023-05-26
No meeting, decision made in the task
== Decision taken ==
Option 2 was chosen.
=== Rationale ===
We opted for a simpler cli that tackles secret and potentially non-secret data by the name of envvars, making sure that there's enough documentation to clear any confusion on the user side on the secrecy of the data.
== Problem ==
Currently users have to store the secrets in the NFS home directory, making it easy to leak (just forget to chmod) and hard to track. This also binds our users to NFS (instead of being agnostic from the implementation).
We want a way for users to create the secrets and not have to care where are they stored, if they set the right permissions and abstracted enough for us to be able to change the implementation with minimal impact on the users.
Original task: phab:T334578
=== Constraints and risks ===
* If it's too complicated users might have a hard time using it
* If we leak implementation details, we might lose the ability to change the implementation
== Options considered ==
== Options ==
=== Option 1 ===
Have a secrets cli (and API) that allows setting secrets as environment variables and files mounted in the filesystem the app runs on (similar to what k8s secrets offers).
Pros:
* Things like certificates are easier to think about (file vs reading it from an env var)
Cons:
* A bit more complex to implement (two features)
* Risk of having users uploading non-secret files as secrets (ex. settings.py)
* No explicit envvars offering (a bit out of scope, but a nice plus)
* If users get used to using secrets for envvars, might difficult implementing envvars in a different way in the future (as users will have to explicitly 
=== Option 2 ===
Have an envvar cli (and API) that allows setting environment variables for the app environment, that would be a superset as env vars would be secrets and not secrets (no specific cli for secrets).
Pros:
* Promotes having only secret things as env vars have size limits
* Simpler implementation
Cons:
* A bit confusing if it is secret or not
* Makes managing things like certificates a bit harder (as there's no file support)
* If users get used to using envvar for secrets, might difficult implementing secrets in a different way (as users will have to explicitly migrate)
=== Option 3 ===
Have a secrets cli and an envvar cli that allow setting environment variables for the app envirnoment, for starters one being just an alias of the other.
Pros:
* Promotes having only secret things as env vars have size limits
* Simpler implementation
* Though they are the same, it makes clear that secrets are supported, and envvars are supported, without mixing them (from the user's point of view) 
* Easy to evolve each offering, allowing splitting the implementation with little user work
Cons:
* Makes managing things like certificates a bit harder (as there's no file support)