nyuuzyou/wikis huggingface dataset:
Fundraising/Data and flow/Database cheatsheet

contribution_status
id	status 	What does it mean
1 	Completed 	
2 	Pending 	
3 	Cancelled 	
4 	Failed 	
5 	In Progress 	
6 	Overdue 	
7 	Fund-to-fund-transfer	
8 	Paid 	
9 	Refunded 	
11	Partially paid 	
12	Pending refund 	
13	Chargeback 	
14	Template 	
This comes from contribution_status in civicrm_option_group (id=11) and then the values from civicrm_option_value
contribution_recur_status
id	status 	What does it mean
1 	Completed 	
2 	Pending 	new default as of February 2021 and a status is not specified
3 	Cancelled 	
4 	Failed 	
5 	In Progress	the state for active recurring charges
14	Processing 	what recurring charge moves to during charge
15	Failing 	
This comes from contribution_recur_status in civicrm_option_group (id=139) and then the values from civicrm_option_value
civicrm_finanical_type
id	name	What does it mean
9 	Cash	default for donations
 	 	
 	 	
civicrm_payment_processor
id	name 	payment_processor_type_id
1 	adyen 	14
2 	amazon 	15
3 	dlocal 	16
4 	globalcollect	17
5 	ingenico 	18
6 	paypal 	19
7 	paypal_ec 	20
 	braintree 	21
 	fundraiseup 	22
This isn't all the payment instruments just the ones I was looking for the ids for so far.
payment_instrument
label 	value
Check 	4
Bank Transfer: PayTM Wallet  	236
Bank Transfer: UPI 	238
Venmo 	274
ACH 	277
This comes from contribution_status in civicrm_option_group (id=10) and then the values from civicrm_option_value
Fundraising/Data and flow/Exchange rates

Uses
- Monthly Convert minimum ask
- Matching gifts in civi uses it wmf_crm.module 306 using the keys to get the currency symbols
- are there more?
Process
They are updated nightly [how] and put into a database in the drupal db named exchange_rates
How to update for SmashPig
On civi1001 run `Drush make-exchange-refs [directory you want it to go into]` for example `Drush make-echange-refs /tmp/`
This creates a file called CurrencyRates.php
Check this into SmashPig, example patch here https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/SmashPig/+/669955
Fundraising/Data and flow/Monthly convert

== Overview ==
Monthly convert is a popup on payments wiki that asks the donor to start an additional recurring donation.
The first donation will go into civi as a one time donation. The donor will get a normal thank you email.
The monthly convert donation will send a new recurring donation to the recurring queue. This will be imported to civi as a new recurring donation with the start date one month in the future and the status as Pending. The donor will get a special monthly convert email.
== Current Payment Methods (as of 2024-08-05) ==
Payment Processor	Payment Method 	Pre Payment Monthly Convert	Post Payment Monthly Convert
Adyen 	Credit/Debit Cards	No 	Yes
Adyen 	Google Pay 	No 	Yes
Adyen 	Apple Pay 	No 	Yes
Adyen 	iDEAL 	Yes 	No
Adyen 	ACH 	No 	Yes
Adyen 	SEPA 	Yes 	No
Braintree 	Venmo 	Yes 	No
Paypal 	Paypal 	Yes 	No
dLocal 	Credit/Debit Cards	No 	Yes (not India)
dLocal 	UPI India 	 	No
== Current Countries (as of 2024-02-14) ==
An up to date list of current config is here: Enabled Monthly Convert Countries
Country 	Code	Added
Austria 	AT 	
Austraila 	AU 	
Belgium 	BE 	
Brazil 	BR 	2024-02-14
Canada 	CA 	
Chile 	CL 	2024-02-14
Colombia 	CO 	2024-02-14
Czechia 	CZ 	
Denmark 	DK 	
Spain 	ES 	
France 	FR 	
United Kingdom	GB 	
Greece 	GR 	
Ireland 	IE 	
Israel 	IL 	
Italy 	IT 	
Japan 	JP 	
Luxembourg 	LU 	
Malaysia 	MY 	
Mexico 	MX 	2024-02-14
Netherlands 	NL 	
Norway 	NO 	
New Zealand 	NZ 	
Peru 	PE 	2024-02-14
Portugal 	PT 	
Sweden 	SE 	
United States 	US 	
Uruguay 	UY 	2024-02-14
== Strings ==
Current English monthly convert strings:
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/DonationInterface/+/refs/heads/master/gateway_common/i18n/interface/en.json
And the descriptions for the strings:
https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/extensions/DonationInterface/+/refs/heads/master/gateway_common/i18n/interface/qqq.json
== Enabling ==
Monthly convert is turned on by country by $wgDonationInterfaceMonthlyConvertCountries in LocalSettings.php
$wgDonationInterfaceMonthlyConvertCountries = [ 'US' ];
here is the ref link for our MC rules:https://meta.wikimedia.org/wiki/MediaWiki:FundraisingBanners/LocalizeData-2017.js
It is turned on by payment method in the Donation Interface adapter with getPaymentMethodsSupportingRecurringConversion
== Deploying ==
This is deployed with the payments-wiki project
== Testing ==
When testing locally, after you make the donation and click yes on the monthly convert the queues need to be consumed in this order:
donations -> contribution tracking -> recurring
To force monthly convert in the url you can add debugMonthlyConvert=true
Fundraising/Data and flow/PSP integrations

= Overview = 
= Capability grid that was on homepage = 
Payment processor capabilities:
 	Ingenico 	PayPal 	Amazon 	Adyen 	D*Local 	Braintree 	Banks 	Checks
Credit card 	Yes 	Yes 	Yes 	Yes 	Yes 	Yes 	 	
Bank transfer 	Yes 	Yes 	No 	Yes 	Yes 	 	IBAN, Swift	
Countries 	 	list 	USAhttps://payments.amazon.com/sdui/sdui/about?nodeId=73479#feat_countries	 	Latam+IN	 	 	
Currencies 	See GlobalCollectAdapter::getCurrencies 	list 	USD 	Allhttp://www.adyen.com/platform/all-countries-all-currencies/	 	USD (different accounts for other curriencies)	 	
Direct debit 	DE, IT, NL (todo: AT, BE, CH, ES, FR, GB) 	No 	No 	Yes 	 	 	 	
Recurring 	Yes 	Yes 	Yes 	Yes 	n/i 	Yes 	 	
Mobile optimized 	No 	n/ihttps://www.paypal.com/uk/webapps/mpp/sell-mobile 	n/ihttps://payments.amazon.com/sdui/sdui/business?sn=devfps/mps 	n/i 	 	 	 	
Languages 	See GlobalCollectAdapter::getAvailableLanguages. Our code must find a fallback language if the donor's native tongue is unsupported.	See PaypalAdapter::stage_locale. For unknown reasons, we have to specify language *by country*.	? 	? 	 	 	 	
Donor needs account 	No 	Yes 	Yes 	No 	No 	Yes 	 	
Refund by API 	n/i 	n/i 	n/i 	n/i 	n/i 	 	 	
Fully automated auditing	Yes 	Yes 	Yes 	Yes 	Yes 	Yes 	 	
Legend:
Yes	Implemented
n/i	Not yet implemented
No 	Unsupported by processor
Fundraising/Data and flow/PSP integrations/Amazon

== Overview ==
Using version 1
Main documentation here:
https://developer.amazon.com/docs/amazon-pay-api-v2/v1-introduction.html?ref=py_apiguide_livecard_sboxhome_US
Useful concept overview:
https://developer.amazon.com/docs/amazon-pay-api-v2/v1-introduction.html
We forked their SDK as it was missing reporting:
https://github.com/ejegg/login-and-pay-with-amazon-sdk-php
== Payment Methods ==
== Recurring ==
== Audits ==
== Testing ==
Test accounts can be made from the sandbox login on sellercentral
== Errors ==
Amazon can have cryptic failmails when they add new IP addresses that we need to add to our allow list. More info at failmail zoo:
Failmail_zoo - Amazon firewall problem
Fundraising/Data and flow/PSP integrations/Gateway Chooser

== Overview ==
The gateway chooser is a special page on paymentswiki used as a middleware to select a payment gateway (aka payment processor) for each donor based on their country, currency, and selected payment method, then prioritize the final gateway base on certain rules. Links from https://donate.wikimedia.org/ and banners will send donors to this special page rather than to the special pages for individual processors, so we can quickly respond to any outages by simply changing settings at paymentswiki.
The gateway chooser examines query string parameters currency, country and payment_method (e.g cc or apple, representing credit card and Apple Pay), recurring (boolean), and other params like payment_submethod as visa amex and so on. It loads yaml configuration files for all enabled gateways to filter down to a list of gateways which support the intended donation parameters.
Then finally it uses rules (defined in extension.json as DonationInterfaceGatewayPriorityRules) to select the target gateway. Each rule can have conditions (if no conditions exist it is the default, that is, it always applied) and a list of gateways in priority order. All conditions must match for a rule to be selected.
== Testing ==
We can go to DI box
docker-compose exec -w "/var/www/html/" payments /bin/bash
then test testAssertExpectedGateway to test all the target gateway
php tests/phpunit/phpunit.php --filter testAssertExpectedGateway
locally we can test as below
https://localhost:9001/index.php?title=Special:GatewayFormChooser¤cy=USD&country=US&payment_method=cc
Fundraising/Data and flow/PSP integrations/Gravy

== Overview ==
We are currently working on integrating with gr4vy using the Secure Fields integration.
== Payment Methods ==
* Credit Card (through Adyen)
* Google Pay/ GPay (through Adyen)
* Paypal
* ACH (through Trustly)
== Console ==
=== What the different recurring panel labels mean ===
Payment source:
card_on_file - in Adyen if monthly convert is enabled one time donations will show this
recurring - this is a recurring payment
Merchant Initiated - can be true or false, if true we initiated it like in a 2+ recurring, otherwise false
Subsequent Transaction - can be true or false, is this the 2+ donation in a recurring
=== Setting up 3DS rules ===
== All the different ids ==
== move the image down ==
== Testing ==
TBC
Fundraising/Data and flow/Payment methods/Apple Pay

Apple Pay is a mobile payment service by Apple Inc. that allows users to make payments in person, in iOS apps, and on the web. 
We have integrations both directly through Adyen and through Gravy using the Adyen backend. There is also native iOS payments that are using the direct Adyen integration.
= Apple Dev Accounts =
We share some Apple Pay setup with the iOS team. Not everyone has access to the full config so if something is needed ask the iOS team and they can help out.
= Adyen =
Adyen was our first Apple Pay integration. We are using Adyen's certificate for setup.
More info here: https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout#Apple_Pay
= Native =
More information here for adyen https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/PSP_integrations/Adyen_Checkout#In_App_(Native)_Donations
= Gravy =
=== Verifying Merchant Domains ===
* Navigate to the Merchant Identifiers list here
* Select the 'Wikimedia Apple Pay Merchant ID' record to edit
* On the Edit or Configure Merchant ID page, Scroll down to the Merchant Domains block
* On the right, click Verify
* Download the new verification file payments.wikimedia.org
* In payments-wiki, update the contents of .well-known/apple-developer-merchantid-domain-association.txt with the contents of the new verification file you just downloaded
* Deploy that change
* Reverify by clicking 'OK'
It will take a few seconds to process, and then if successful, you will be redirected back to the Edit or Configure Merchant ID page where Verification Expires date should be updated.
Fundraising/Data and flow/Pending transaction resolver

= Overview =
The pending transaction resolver (PTR) is a job that takes care of donations which were left authorized but not captured at payments-wiki. This may be because the fraud score was high enough that we didn't immediately capture the donation, or because the donor didn't make it back from a processor's hosted page to payments-wiki.
Payments-wiki sends information about donation attempts to a 'pending' table, generally before redirecting donors to a hosted payment page. When a donation attempt definitively succeeds (payment captured) or fails (authorization rejected) we delete entries from this table. The rest of these attempts stay in the table until the PTR deals with them.
The PTR examines pending table entries one by one, first getting the latest status and examining the fraud scores. When fraud scores are below the review threshold, the PTR simply tries to capture the donation. Otherwise the PTR checks to see if the name and email in the pending row match an existing donor with no refunds (there is no time limit on donations or refunds). If they match, the PTR tries to capture the donation. In either case, the PTR will NOT try to capture a donation when the name and email match anyone with a successful donation in the past day.
= Adyen =
The Adyen resolver runs every 30 minutes with the job adyen_pending_resolver.yaml 
The job calls cvapi PendingTable.consume version=4 gateway=adyen
= Ingenico =
The Ingenico resolver runs every 30 minutes with the job ingenico_pending_resolve.yaml
The job calls cvapi PendingTable.consume version=4 gateway=ingenico
= Diagrams =
Our diagrams on gitlab https://gitlab.wikimedia.org/repos/fundraising-tech/fr-tech-diagrams/-/tree/main/pending_transaction_resolver
= Testing =
drush cvapi PendingTable.consume version=4 gateway=adyen batch=1
This will run the resolver once locally.
Fundraising/Data and flow/Queues/Queue performance log

Tue 01 Nov 2022 11:58:53 PM UTC
Queue speed in 3 hour test was slow.
1) Redis cache hits still seemed higher than I expected - I created an issue to add some notes on re this 
2) There were high deadlocks. The deduper job was running, which I thought should have been throttled. On looking the throttling depends on a drush variable threshold which stands for 'max number of contacts created in the last (5) minutes for the job to be permitted to continue. I checked what the threshold was and found it was not set. Until the threshold_period the threshold has no default value. This meant the deduper throttling was effectively turned off.
I set the threshold to be 250 (drush @wmff vset threshold=250) - so if 250 contacts have been created in the last 5 minutes deduper will not run. I don't know if that threshold is 'right' - we update more often than we create. Christine did suggest changing to 'modfied_date' which might be better, although it would pick up bulk changes that are not ongoing. We can monitor... however, the goal is that the deduper job should not run during time of high load & it was running
Fundraising/Data and flow/Stats pipeline

== Banners ==
Data about displays and interactions with banners mostly comes via two background requests made by users' browsers when users visit one of the production wikis where banners may be shown. The requests are: beacon/impression and banner history. Data from these requests passes through multiple systems and ends up in multiple data stores, in multiple formats.
=== beacon/impression ===
This is the main data point for CentralNotice activity on each pageview. This request is sent on 100% of pageviews for Fundraising campaigns, and is usually sent for only 1% of pageviews on non-Fundraising campaigns.
Full data can be extracted from the Hive webrequest table. Aggregated data is stored in the Druid banner_activity_minutely datastore. On the Fundraising cluster, the pgehres bannerimpressions table also stores aggregated data, generated from a 10% sample of the requests received from browsers.
=== Banner history ===
CentralNotice's banner history feature is typically enabled for all Fundraising campaigns. When a user is targeted by a campaign with this feature enabled, so long as the user does not have the Do not track preference set in their browser, CentralNotivce stores in the browser's LocalStorage a log of CentralNotice activity, with entries for each of the users's pageviews over the course of the campaign.
For Fundraising campaigns, CentralNotice is normally configured to send summary versions of these logs back to WMF servers on 1% of pageviews targeted by the campaign. In addition, if a user clicks a Fundraising banner to donate (that is, if they click the button that will take them to the form where they enter payment details) the log is sent 100% of the time.
Banner history logs are sent via the EventLogging system. Here is the current schema.
=== Impression rates ===
See centralnotice_analytics.
== Landing pages ==
== Older Content moved from main page == 
====Banner Impression/Landing Page Stats Collection====
Banner impressions and landing page stats are collected from the production proxies. Fundraising_Analytics/Impression_Stats. The wmf:Thank_you page includes wmf:Template:Hide_banners which loads Special:HideBanners from multiple domains via image src. HideBanners sets cookies for donors which tell CentralNotice's bannerController.js not to pester them for a year or so.
==== utm_source====
This is a tracking variable which is supposed to collect information about the transaction. Currently, it is a period-separated concatenation of three components. One interpretation of the components is, 1) banner name, 2) landing page name, and 3) payment method. We are currently in the process of standardizing (see FR #965 and FR #673).
In theory, each component may be a tilde-concatenation of a sequence of landing pages, for example. That code is badly dysfunctional.
====utm_medium====
Donor was referred by this type of site: sitenotice, spontaneous, sidebar, socialmedia.
Seems unuseful at this broad granularity.
====utm_campaign====
The parent campaign for the banner where this donation was initiated.
====utm_key====
TODO
Fundraising/External-facing/CiviProxy

== CiviProxy On Docker ==
=== Overview ===
As part of the email preferences work relating to this ticket, we added Civiproxy to the fundraising tech docker stack.
== Setup ==
CiviProxy should work under docker-compose after running setup.sh. Here are some of the things we had to do to get it running:
=== Hard-code CIVICRM_SITE_KEY to docker_site_key ===
CiviProxy and CiviCRM need to have the same value set for their site key to let CiviProxy make requests to CiviCRM. 
We set an environment variable at the top level in docker-compose.yml: 
 x-civicrm-site-key:
 &civicrm-site-key FR_DOCKER_CIVI_SITE_KEY=docker_site_key
The we pass it through to the civiproxy and civicrm containers:
 civiproxy:
 environment:
 - *logger-host
 - *civicrm-api-key
 - *civicrm-site-key
In the civibuild wmff install.sh we pass through the environment variable to set a local env variable that is used by Civi setup so it skips generating a random key.
In our CiviProxy public config (checked into the fundraising-dev repo) we pass that env var through to the $sys_key_map:
 $sys_key_map = array('SITE_KEY' => getenv('FR_DOCKER_CIVI_SITE_KEY'));
=== Update `civicrm`.`contact` api_key field for admin user to 'docker-api-key' ===
The two systems also need to agree on an api key that can be used by CiviProxy to authorize calls to CiviCRM. We automated this setup by adding an API call to the install.sh script in civibuild's wmff config. See a screengrab of why it was necessary. 
== Other ==
=== Docker stack CiviProxy Test URL ===
https://localhost:9005/rest.php?entity=civiproxy&action=getpreferences&key=SITE_KEY&version=3&json=1&api_key=API_KEY&hash=256fa6f5a00bce4598bc9144f203132f&contact_id=204
=== Gerrit patches ===
* dev-images: (only needed if building locally): https://gerrit.wikimedia.org/r/c/releng/dev-images/+/664919
* Fundraising-dev: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/dev/+/673094
* CiviCRM: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/649729
=== Local Troubleshooting ===
See Fundraising/External-facing/E-mail_preference_center#Local_Troubleshooting
Fundraising/External-facing/Payments

= Overview =
= Upgrading MediaWiki =
When a new LTS (Long term support) version of MediaWiki comes out, we try to upgrade Payments-wiki to use it fairly quickly. To upgrade from our customized REL1_35 branch (fundraising/REL1_35) to a customized REL1_39 branch, we follow these steps:
* Create the new fundraising/REL1_39 branch of mediawiki/core as a merge target in Gerrit. This should be based on the REL1_39 branch.
* Pull the new fundraising/REL1_39 branch down locally
* Remove extensions and skins that we don't deploy (patch from last time: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619814)
* Add logos (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619815)
* Add payments-specific extensions (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/619817)
** All of the core extensions and skins that we use on payments-wiki should be checked out at REL1_39. DonationInterface should be checked out at deployment, and FundraisingEmailUnsubscribe at the master branch.
* Update composer.json to merge extension composer deps and point to our amazon SDK fork (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/620118/3/composer.json), and pin php version to prod php version (e.g. https://gerrit.wikimedia.org/r/c/mediawiki/core/+/634577).
* Create the new fundraising/REL1_39 branch of mediawiki/vendor as a merge target in Gerrit. This can be based on the REL1_39 branch, but it's not so important, as we will clear it out in the next step.
* Install dependencies into vendor with --no-dev, check the changes in to the fundraising/REL1_39 branch of vendor, and update the submodule pointer from the fundraising/REL1_39 branch of core
** On your first try, this may come up with some of conflicts if DonationInterface's composer.json has some requirements pegged to the versions supported in the previous mediawiki version. Update those requirements to transitionally support both the previous MW version and the new MW version.
* Run php maintenance/update.php on your local installation and note any changes needed in LocalSettings.php
= Moved from main page =
===Translations===
See Fundraising/Translation for more info
*Donatewiki translations go out regularly on the l10n cache
*TYs need to be manually deployed - make a task for this and put it in pending review in the current sprint
* Subject line needs to be manually deployed - make a task for this and put it in pending review in the current sprint
*Payments needs to be manually deployed - make a task for this and put it in pending review in the current sprint
Fundraising/Internal-facing/CiviCRM/Exporting FormBuilder Pages into an extension

Like the SearchKit Search export, this would also be helpful if the extension by which the FormBuilder page is built on was designed to be reusable. The following steps would describe how to ensure a FormBuilder page is packaged into a custom extension.
# Having completed the designing of the FormBuilder page.
# On the FormBuilder home page, locate the row that corresponds to your FormBuilder page and take note of the designated name in the name column.
# On the computer file system, navigate to this directory: drupal/sites/default/files/civicrm/ang. This directory is where the Afform templates are initially stored whenever the FormBuilder page is designed.
# To make this FormBuilder page available wherever the extension is installed, the files corresponding to the desired FormBuilder page would have to be moved to the extension directory.
# Search for files corresponding to the name of the FormBuilder page (from Step 2). Assuming the name of the FormBuilder page is afsearchTest, the required files to copy in the directory would be afsearchTest.aff.html and afsearchTest.aff.json.
# Copy these files.
# On the extension direction in the file system, create a new directory titled "ang".
# Paste this new files into the "ang" directory created in Step 7.
# Now the FormBuilder page would always be available wherever the extension is installed.
====== Related Topics: ======
* Exporting SearchKit Search into an extension
* Create Custom Formbuilder pages to monitor tables in other WMFF Databases
* Creating extensions in Civicrm to manage tables in other WMFF Databases