nyuuzyou/wikis huggingface dataset:
Fundraising/Team processes/DRI

= Designated Response Individual(s) (aka The Chaos Crew) =
In August of 2022, the Fundraising Tech (FR-Tech) team began incorporating a Designated Response Individual (DRI) model to the team's workflow.
The folks currently tasked with being the DRIs are affectionately known as the Chaos Crew.
== What is DRI? ==
For each sprint, there are 2 individuals designated as the DRIs; a primary and secondary.
These two individuals are responsible for fielding FR-Tech's interrupt and reactive work.
The goal is to reduce the interruption on the engineers doing the feature-driven sprint work.
== General Responsibilities ==
=== In scope activities ===
* Incident response, management, and retrospectives
* Failmail/Fraudmail
* Chores list - 
* When time available:
** Proactive investigations into sustainability work, eg: performance, testing, technical debt.
=== Out of scope activities ===
* Major refactors the should be scoped, planned, and reviewed.
== Processes ==
* Task tracking is done via "Fundraising Tech - Chaos Crew" workboard board.
* The workboard is a long-running Kanban board (not a sprint board).
* Tasks are triaged (prioritized) as they come in. If not, they are triaged on a weekly basis following the regular Monday backlog meeting.
* During the Sprint/Chaos Crew change-over meeting, the Chaos Crew does a Chaos crew handover.
== Rotation ==
Everyone rotates every 2-week sprint:
* Engineer n -> Engineer n+1 (no effective role change)
* Engineer 4 -> Secondary DRI
* Secondary DRI -> Primary DRI
* Primary DRI -> Engineer 1
A full rotation occurs every 6 sprints, which is about every 3 months, so about every quarter.
See: Current rotation assignments.
== DRI Roles ==
There are two DRI individuals: a primary and a secondary. 
=== Secondary -> Primary roll-in ===
When someone transitions from feature work to DRI work they start as the secondary. This is to allow any incidents that began during the previous sprint to have continuity by the now-primary DRI. The secondary can then get up to speed from the primary.
=== Primary DRI responsibilities ===
* Primary incident response.
* Shares relevant updates with Secondary DRI.
* Proactive investigations
=== Secondary DRI responsibilities ===
* Transitioning from feature work duties to DRI duties,
** eg: final code-reviews, hand-offs, or similar.
* Reviewing the activities of the DRI Team’s previous sprint including any active or on-going issues.
* Chores list.
=== Primary & Secondary DRI responsibilities ===
* Automation!
** Especially the chore work.
* Documentation!
** If you answer a question/resolved an issue that wasn’t already documented, document it!
** “Wiki gnome / wiki gardening” activities appreciated.
* Log DRI Team activities on the tracking task.
== References ==
* Microspeak: DRI, the designated response individual
Fundraising/Team processes/DRI/Chores

==Checking damaged messages==
https://civicrm.wikimedia.org/damaged
* In CiviCRM, go to /damaged
* Look at the stuff from this week
* If the error looks like a database lock, please requeue the message
* If it's a new type of failure, please create a phab ticket
==Following up on incoming failmail==
* Update the Failmail zoo if it's something common
* File phab tasks with details of what we need to do so it stops failmailing.
* Email team letting them know it's been documented
==Follow up or ack incoming alerts (from payments team or processors) to fr-tech@ mailbox==
==Log checking==
On our centralized logging box, scan through the error logs from today
* Create phab tasks for any errors that are unexpected/new 
* (errors generally should not be expected)
* https://github.com/jackgleeson/binscripts/blob/master/logdog.sh
==Deployments==
* CentralNotice (not for long)
* CiviCRM
* Payments-wiki
* SmashPig
==Check on audit incoming directories==
* If they are getting really full, figure out why
* Either move out stuck files or run in --make-missing mode
==Civicrm Upgrades==
==Docker maintenance and updates task creation==
==Every 90 days should login to test consoles for all payment gateways to prevent lock account==
We have accounts that log us out after a certain amount of inactivity, log into them when you join and leave rotation.
* Ingenico dev enviornment - 90 days - https://wpc.gcsip.nl/wpc/login
* Ingenico production environment - 90 days - https://wpc.gcsip.com/wpc/login
Fundraising/Team processes/Definition of Done

A definition of done is a way for our whole team to agree on our habits for pushing out quality code. All of these things need to be done before a phabricator task can be marked as complete by the stakeholder, fellow engineer, or product owner.
During a sprint, a task moves between several columns in a sprint board. A task is totally done when it is in the "Done" column and in a "resolved" state. The following must be true to fully resolve the task:
#Steps to done or to replicate - If the done state is not clear, it shouldn't be in a sprint or started. Clear path for manual testing written up on the Phabricator ticket.
#Tests written - If not, comments about lack of test coverage. Code review to confirm (task placed in review column).
# Code review - Build succeeds and reviewer has followed steps to done, code should be understandable and, if necessary, should include explanatory comments (when review is complete, task moves to "Pending deployment").
# Documentation - a link to a Mediawiki, Wikitech (if applicable only to our install/deploy), Collab (if super confidential) page that will describe expected behavior in the Phabricator task.
#Works in production according to requirements. Reporter of bug (unless otherwise stated) to confirm this and close the task. 
There can be exceptions (for example urgent bug fixes, tight deadlines). Also some tasks are purely technical and can be resolved by another engineer.
Fundraising/Team processes/Mid-Sprint Checkup

== Sprint Checkup ==
 
Our mid-sprint reviews are used to reflect and focus on what is to be done for the remainder of the sprint. Sometimes we will reprioritize but mostly it is use to refine and make sure we can meet our sprint goal.
This meeting should be succinct and complete within 20-30 minutes.
== Checkup Agenda ==
* Check up on Ops topics for alignment and opportunities for cross team collaborations.
* Review the agreed upon sprint goal (in the form of a sprint goal ticket)
* Reflect and focus on what is in the sprint
** State the remaining time in the sprint and any upcoming scheduling/vacation/holiday/other issues
** Review the overall state of the board
** Is there anything that needs to be moved to a new column? (not always explicitly acked)
** Is there anything that is blocked?
** Does anyone need help?
** Is there anything we know that we can not get accomplished in the remaining sprint time? Remove any tasks from the sprint that are unattainable.
** Does anyone feel like they are running out of things to do or needs direction?
* Review the DRI/Chaos Crew board
** Is there anything that needs feature team eyes?
** Anything of import that would affect current sprint work?
** Is there anything that needs cross team work (ops/other)?
** If needed, prioritize any tasks in the Chaos backlog column.
Fundraising/Team processes/New integration manual

This is the development process to integrate with a new payment processor.
== Preliminaries ==
* Review documentation and confirm that the payment processor seems to comply with our technical, security, and communication standards.
* Create test accounts.
* Request production accounts; give us plenty of back-and-forth time with getting those setup.
* Figure out the API endpoints and begin a discussion about stable IP address ranges for these servers, so we can add to the firewalls.
== First milestone: ready for internal test ==
When this milestone is completed, we will be able to deploy code to staging or production which demonstrates some basic payment workflow. Fundraising and tech team members will be able to donate, along with selected testers in the target countries. We should not expose the page to the public at this point.
What we need for internal testing:
* Working frontend form.
* Backend processing workflow is able to make a charge and push the message onto the queue.
* Tech team gives fr-online low-amount test links for target country / currency / language combinations
What we can skip for this step:
* Audit processing. We can always pick these files up and import in the future.
* Listener. Some workflows rely on the listener to finalize the payment, in this case we usually want to include that in the 1-hour test.
* The queue message can't break the consumer. Not strictly necessary that we can import it correctly, however, this is sort of a grey zone.
Be sure not to publish the new form at this point. You will want to add the form definition in DonationInterfaceFormSettings.php, but set the selection_weight to 0 so that the form cannot be chosen without specifically naming it in the URL. We don't want donors accidentally navigating to the form.
== Second milestone: 1-hour donor test ==
We will have a frontend stable enough to:
* Provide decent donor user experience.
* Process audit files and potentially listener messages, in order to send the donors their thank-you letters in a timely fashion. Ideally, this is a 100% thing, otherwise Tech and Donor Services will have to play catch-up.
* Process payments in a manner that makes it possible for Donor Services to easily find, modify, and refund one-time and recurring payments in the payment processor console.
== Final milestone: campaign-ready ==
We have iterated with 1-hour tests until we're certain that the bugs are ironed out. We follow the Campaign Readiness Checklist for consistency.
Fundraising/Team processes/On-call guide

[WIP]
== Before on-call duty ==
* Review this documentation.
* Verify your contact info on Office Wiki and Collab, and update if necessary.
* Verify your ability to log in to external and internal services, including access to 2FA tools required to do so.
* Update and verify your local dev environment.
== During on-call duty ==
* Keep your work laptop, Yubikey and 2FA devices with you.
* Stay in a place with reliable internet, or where you can get to reliable internet quickly.
* Keep your phone charged, with notifications/ringer volume on.
* Check e-mail, IRC, Slack and Grafana regularly.
* Remain relatively lucid.
* If an emergency occurs and you're not sure what to do, reach out to others.
* Communicate any actions you take via e-mail, IRC and/or Slack, as appropriate.
== On call schedule 2024 ==
End of Year 2024 on call rotation
== Fail Mail ==
Fundraising/Data and flow/Failmail zoo
== How to take down banners ==
Instructions for taking down banners
== Other documentation ==
[TODO review and possibly update or include contents from these documents here]
https://office.wikimedia.org/wiki/Fundraising/Engineering/On_call
https://collab.wikimedia.org/wiki/Fundraising/Engineering/Shutting_the_pipeline_down
https://collab.wikimedia.org/wiki/Fundraising/Engineering/Fundraising_campaign_takedown
https://collab.wikimedia.org/wiki/Fundraising/Engineering/Outages
https://collab.wikimedia.org/wiki/Fundraising/Engineering#Emergency_Protocol_/_Incident_Response
https://office.wikimedia.org/wiki/Fr-tech_Emergency_Contact
https://collab.wikimedia.org/wiki/Fundraising#Production_Emergency_Contacts
https://collab.wikimedia.org/wiki/Fundraising/PSP_contact
https://collab.wikimedia.org/wiki/Fundraising/Engineering/Disaster_Protocol
https://collab.wikimedia.org/wiki/Fundraising/Engineering/Emergency_Datacenter_Cutover_Procedure
Fundraising/Team processes/Training

Person/Project knowledge table.
 	eileen	ejegg	cstone	jgleeson	damilare	wfan	dwisehaupt	Other (team/individuals)
Core Dev 	 	 	 	 	 	 	 	
Docker 	 	X 	X 	X 	X 	 	 	
Continuous integration jobs 	X 	X 	 	X 	 	 	 	Release Engineering
CiviCRM 	 	 	 	 	 	 	 	
CRM - Civi 	X 	X 	X 	X 	X 	X 	 	
CRM - Database 	X 	X 	X 	X 	X 	X 	X 	
CRM - Queue consumers 	X 	X 	X 	X 	X 	X 	 	
CiviProxy 	 	 	X 	X 	X 	 	 	
Redis 	 	X 	X 	X 	X 	X 	X 	
MediaWiki Related 	 	 	 	 	 	 	 	
CentralNotice 	 	X 	X 	X 	X 	X 	 	
Donate Wiki / FLP 	 	X 	X 	X 	X 	 	 	Peter Coombe
DonationInterface antifraud 	 	X 	X 	X 	X 	X 	 	
DonationInterface backend 	 	X 	X 	X 	X 	X 	 	
DonationInterface frontend 	 	X 	X 	X 	X 	X 	 	
Email Preference Centre 	 	 	X 	X 	X 	 	 	
SmashPig 	 	 	 	 	 	 	 	
SmashPig listeners 	 	X 	X 	X 	X 	X 	 	
SmashPig job runners 	 	X 	X 	X 	X 	X 	 	
Payment Integrations 	 	 	 	 	 	 	 	
Payments deployment 	 	X 	X 	X 	X 	X 	 	
Adyen 	 	X 	X 	X 	X 	X 	 	
Amazon 	 	X 	X 	X 	X 	X 	 	
D-Local 	 	X 	X 	X 	X 	X 	 	
Braintree 	 	X 	X 	X 	X 	X 	 	
Ingenico 	 	X 	X 	X 	X 	X 	 	
Paypal Express 	 	X 	X 	X 	 	X 	 	
FundraiseUp 	 	 	 	 	X 	 	 	
Auditing 	 	 	 	 	 	 	 	
Auditing - Amazon 	 	X 	 	X 	 	 	 	
Auditing - Adyen 	 	X 	X 	X 	X 	X 	 	
Auditing - Ingenico 	 	X 	X 	X 	X 	X 	 	
Auditing - Braintree 	 	X 	X 	X 	X 	X 	 	
Auditing - PayPal 	 	X 	X 	X 	 	 	 	
Auditing - FundraiseUp 	 	 	X 	X 	X 	 	 	
Analytics 	 	 	 	 	 	 	 	
Analytics infra 	 	 	 	 	 	 	X 	
Banner imps - Old (UDP2log/pgehres)	 	X 	 	X 	X 	 	 	
Contribution tracking 	X 	X 	X 	X 	X 	X 	 	
Fredge 	 	X 	X 	X 	X 	X 	 	
Hive 	 	X 	 	 	 	 	 	
Email 	 	 	 	 	 	 	 	
Email unsubscribe 	 	X 	X 	X 	X 	X 	 	
Acoustic 	X 	X 	X 	X 	X 	X 	 	Peter Coombe
Fundraising Tools 	 	 	 	 	 	 	 	
Silverpop Export 	 	 	X 	X 	 	 	 	
Misc 	 	 	 	 	 	 	 	
Process-Control Jobs 	 	X 	X 	X 	X 	X 	X 	
Process-Control App 	 	 	 	X 	 	 	 	
PrivateBin 	 	 	 	 	 	 	X
Fundraising/tech

This page is an overview of Fundraising Tech projects. To find the most recent documentation visit the Fundraising tech page on mediawiki.
== Components ==
Moved to mw:Fundraising_tech/Components
== Fundraising Deployments ==
See fundraising cluster deployments.
== Installing Software ==
There are two ways to go about provisioning your dev environment, the extremely painful yet instructive manual method, and the fun way. Both are documented more fully at mw:Fundraising_tech.
The fun way is to check out our docker repo at 
gerrit.wikimedia.org:29418/wikimedia/fundraising/dev and run setup.sh.
Please develop with E_STRICT.
== See also ==
Category: Fundraising
Category: Fundraising - Needs Updated
Fundraising/tech/Alerts

=== Self-recurring checks ===
11:09 <+Jeff_Green> no. it's a two part thing
11:09 <+Jeff_Green> part one is a perl script collect_activemq_stats_via_gmetric
11:10 <+Jeff_Green> that script is in puppet, it calls 
 silicon:/usr/local/apache-activemq/bin/activemq-admin
11:11 <+Jeff_Green> and parses the output for a whole bunch of predefined queues
11:11 <+Jeff_Green> then it injects the results to ganglia using /usr/bin/gmetric
11:12 <+Jeff_Green> finally it writes a statefile to /tmp/activem_queues.state
11:12 <+Jeff_Green> second part is a nagios plugin
11:12 <+Jeff_Green> silicon:/usr/lib/nagios/plugins/check_activemq
11:14 <+Jeff_Green> that reads the statefile from the ganglia collector, parses for whichever 
 queue you've requested, decided whether or not it is alertworthy (from 
 command line args, per usual nagios plugin usage), and sends the report as 
 a passive check to nagios
11:15 <+Jeff_Green> so the whole icinga part is working but somehow the nagios config is missing
11:15 <+Jeff_Green> all of this is puppetized fwiw
Fundraising/tech/Apache httpd

== Apache httpd ==
You may already have Apache httpd 2.2 already installed.
Check by going to the CLI and typing:
httpd -v
You should get output like:
Server version: Apache/2.2.21 (Unix)
Server built: Nov 15 2011 15:12:57
You will need to enable name based virtual hosting to follow the fundraising tutorials.
You can append something like this to the bottom of your httpd.conf file:
NameVirtualHost *:80
<VirtualHost _default_:80>
 DocumentRoot "/www/sites/localhost/default/public"
 <Directory "/www/sites/localhost/default/public">
 <IfModule mod_dav.c>
 DAV Off
 </IfModule>
 AllowOverride None
 Order allow,deny
 Allow from all
 </Directory>
</VirtualHost>
# Virtual Hosts
Include /www/conf/httpd/sites/*.conf
<IfModule php5_module>
 AddType application/x-httpd-php .php
 AddType application/x-httpd-php-source .phps
 <IfModule dir_module>
 DirectoryIndex index.php index.html
 </IfModule>
</IfModule>
Category:Fundraising
Fundraising/tech/Coinbase

== Import ==
Currently, we upload the reports by hand. Here is how to do so:
* Log into WMF's CiviCRM.
* Go to /admin/import_checks, there should be an "Import Checks" menu item in the left sidebar.
* Click the "Browse" button and select your report file.
* Select "Coinbase" from the file type radio buttons.
* Click "Upload".
The results of your import should appear in the log pane.
== Questions ==
* What is the format for downloading transactions from Coinbase? We will be using their built-in Transaction History and Merchant History downloads? We need a sample of the actual data before development can begin. Let's avoid any custom spreadsheet download if possible.
** If we are committed to a custom spreadsheet for some reason, I have a lot more questions.
* Downloads are nightly? Can we automate using sftp?
* We'll create a new gateway, "coinbase", but should the payment instrument be "cash"?
* Refunds
** Our plan is to have donor services perform refunds in their console?
** How will we identify transactions to refund? Bitcoin http://cryptodetail.com/satoshi-nakamoto-and-suspected-people/ What is Bitcoin and who is Satoshi Nakamoto? transactions can come from multiple sources, for example, so do we pay out to each source individually, or is this transparent to us?
** There was some talk about making bank transfer refunds. Can we agree that we will not do this?
* Recurring
** If a donor asks us to cancel their subscription, it sounds like we only have one option: tell them to cancel themselves in Coinbase. Or can we do that for them?
== Answers ==
=== Schedule ===
We are not releasing until testing is completed.
=== Refunds ===
We will refund in USD, an amount equal to what we received after instant conversion. This is possible through the web UI or by API.
Fundraising/tech/Currency conversion sample

{
 "base": "USD",
 "date": "2021-11-10",
 "rates": {
 "JPY": 0.013125,
 "BGN": 0.6707,
 "CZK": 0.05190,
 "ARS": 0.2294,
 "AUD": 1.0689,
 "CHF": 1.1154
 }
}
Fundraising/tech/Transformer

This is a draft specification for a data transformation component to be used throughout Fundraising.
== Background ==
Many Fundraising libraries need to perform similar transformations between internal (normalized) and 3rd-party API representations of donor and donation data. Wherever possible, we want to extract the gateway-specific transformations into their own component, so that these transformations are applied in a consistent way across codebases. Ideally, the transformation would be described using declarative data except in the few cases where we need custom code.
Currently, this code will be found in DonationInterface, crm, audit scripts, and listeners.
== Implementation ==
=== FieldTransformation ===
A first-class object that describes how to map a small unit of data from one system to another. Try to keep each of these as small as possible, for reuse. This may map a field 1:1 between the systems, perform validations, or do 1-many, or many-1. It is passed all the data in the input array, and the output array in whatever state it has reached.
API:
* run($in, &$out), throws ValidationException
=== Transformer ===
A container holding a list of FieldTransformations. A Transformer object is usually obtained from the gateway, and is specific to a mapping from one system to another. We probably want a factory interface.
=== Request Mush ===
This is ill-defined at the moment. We realized there is a need for gateway-specific logic that can combine GET, POST, and SESSION data using precedence rules. We think it's best if this logic is run on normalized data only.
=== Data System ===
Examples of data systems: 'user:ccform', 'api:DoPayment', 'crm', 'audit:TrrFile', 'normal', 'listener'...
Fundraising/tech/access

Fundraising cluster access, for CiviCRM and other services, generally follows the same best practices as production shell access, including requirements of C-Level approval and a 3 business day waiting period to both allow SRE to schedule the work and to ensure there is sufficient time for any concerns to be raised. This access is highly sensitive and the SRE team is diligent about maintaining security. Accounts inactive for 90 days will be disabled per PCI requirements.
== How to request access ==
* create a Phabricator account if you do not already have one
* create a task with a description of the access you need and a short explanation why, and tag it with 'fundraising-tech-ops'
* obtain C-Level approval and provide proof of it in the task
After 3 business days you will be contacted by a member of the fundraising SRE team.
Fundraising/tech/crm/Add a New Gateway

==Adding a New Payment Gateway==
If you add a new payment gateway in DonationInterface, you'll have to add it a few places in CRM too:
* Add gateway name to custom reports
** crm/modules/wmf_reports/CRM/Report/Form/Contribute/GatewayReconciliation.php
* Add gateway-specific payment instruments
** crm/modules/wmf_civicrm/wmf_civicrm.install, see wmf_civicrm_update_7026
** crm/modules/wmf_civicrm/wmf_civicrm.module, function wmf_civicrm_get_message_payment_instrument
Fundraising/tech/crm/Quirks

Potential frustrations when setting up a dev instance of our CiviCRM setup:
* Need to include vendor/autoload.php from inside of sites/default/settings.php, which is .gitignored (Why did I think this was the spot for it?)
* Gift Data custom fields may be non-functional after running wmf_civicrm_update_7022 from wmf_civicrm.install. Just disable them via the UI ( Administration->Custom Data) TODO: add some option values and groups too, and set option_group_id in the CustomField create calls.