Fundraising/Data and flow/PSP integrations/Adyen Checkout

== Overview == 
Adyen is one of our credit card processors.
In 2021, we updated our Adyen code to use their 'Checkout' integration, version 4. Checkout integration sources Adyen-hosted JS on our payments page and renders payment fields in the same form as the donor data, encrypting the data before submitting it to our server. We are using the 'components' style Checkout integration rather than the 'drop-in' style. Both styles use the same Adyen JS objects, but with components we instantiate a slightly lower-level JS object that is specific to the payment method.
A month after we deployed our new integration, Adyen released Checkout version 5 with some breaking changes and a simplified control flow. We have not made plans to update to version 5.
All calls to the Adyen API are routed through the SmashPig library. There are PaymentProcessor subclasses for each payment method.
Documentation for the former (Hosted Payment Pages) integration is at https://www.mediawiki.org/wiki/Fundraising_tech/Adyen
== Payment Methods ==
=== Credit Cards ===
https://docs.adyen.com/payment-methods/cards/web-component
Credit cards are available with payment_method=cc which adds Adyen's fields (Card number, Expiry date, and CVC / CVV) below ours in the payments form.
Credit Card form
==== 3DS ====
https://docs.adyen.com/online-payments/3d-secure
When 3DS is enabled we collect more information about the donation and send it to Adyen in the browserInfo parameter.
https://docs.adyen.com/online-payments/3d-secure/redirect-3ds2/web-component
Example browserInfo:
 "browserInfo":{
 "userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/70.0.3538.110 Safari\/537.36",
 "acceptHeader":"text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/webp,image\/apng,*\/*;q=0.8",
 "language":"nl-NL",
 "colorDepth":24,
 "screenHeight":723,
 "screenWidth":1536,
 "timeZoneOffset":0,
 "javaEnabled":true
 },
Adyen will determine on their end if the 3DS redirect is needed and if so return an action object on the response. This contains information on the 3DS redirect.
https://docs.adyen.com/online-payments/api-only/handling-redirects#3d-secure
=== iDEAL ===
https://docs.adyen.com/payment-methods/ideal/web-component
iDEAL is bank transfer payment method only available in the Netherlands (payment_method=rtbt&country=NL). 
The supported banking systems can be down so Adyen handles creating the list of available banks (referred to as issuers by Adyen) which we get by calling getPaymentMethods.
The donor chooses a bank and is then redirected to their bank's website to complete the donation. 
When that is complete they are redirected to our thank you page.
 	
List of available banks (issuers)	Example redirect to donors bank
==== iDEAL Recurring ====
https://docs.adyen.com/payment-methods/ideal/web-component#recurring-ideal-payments
iDEAL recurring has a unique flow as Adyen sends the recurring token on an IPN after the donor finishes the transaction. The donation isn't immediately put onto the donations queue.
From the donor's perspective the flow is exactly the same as a one time donation but on our end we don't complete the donation until we get a RECURRING_CONTRACT IPN that has the recurring token on it.
RECURRING_CONTRACT IPNs are set for every new recurring so we filter out just the iDEAL recurrings, find the matching pending database entry, add the recurring token to the message and send it to the donations queue.
Recurring iDEAL donations are of the type SEPA Direct Debit.
p.s. iDEAL do not need separate capture, but the recurring iDEAL is using SEPA, and have the config from Adyen side that control if auto capture or no.
=== Apple Pay ===
* Implemented using https://docs.adyen.com/payment-methods/apple-pay/web-component
* Using "own certificate" approach
Apple Pay is available on Safari with payment_method=apple
It replaces the credit card form with a Donate to Apple Pay button as seen below. When the donor clicks on it it brings up a sheet (I think this is what its called?) that lets the donor choose which card and contact information to use.
 	
Using Apple Pay in Safari	Choosing card and contact information
==== In App (Native) Donations ====
We have two endpoints to handle in app donations, getPaymentMethods and submitPayment
=====getPaymentMethods=====
This takes in the country and returns the available payment methods. The app caches this data for ~1 month.
params
* country (two letter country code - US)
/api.php?action=getPaymentMethods&format=json&country=US
Returns:
* response (all the payment methods available for that country)
Example response: country=US
{"response":{"paymentMethods":[{"brands":["visa","mc","amex","discover","cup","maestro","diners","jcb"],"name":"Credit Card","type":"scheme"},{"brands":["visa","mc","amex","discover","maestro"],"configuration":{"merchantId":"000000000201454","merchantName":"WikiMedia"},"name":"Apple Pay","type":"applepay"},{"issuers":[{"id":"jp","name":"Apple Pay"},{"id":"ap","name":"Google Pay"},{"id":"cs","name":"\u010cesk\u00e1 spo\u0159itelna"},{"id":"kb","name":"Komer\u010dn\u00ed banka"},{"id":"c","name":"P\u0142atno\u015b\u0107 online kart\u0105 p\u0142atnicz\u0105"},{"id":"dpcz","name":""}],"name":"Online Banking CZ","type":"onlineBanking_CZ"},{"configuration":{"merchantId":"50","gatewayMerchantId":"WikimediaDonations"},"name":"Google Pay","type":"paywithgoogle"}]}}
===== submitPayment =====
This takes in a token generated in the app as well as the rest of the donation parameters and creates the payment.
Params
* amount
* app_version
* banner
* city
* country
* currency ( USD, CAD, etc)
* donor_country (2 letter code US)
* email
* first_name
* full_name (we can handle this on our end if the separated name fields arent available)
* language
* last_name
* opt_in (If they agreed to emails or not , 0 or 1 )
* pay_the_fee (if they agreed to pay the fee, 0 or 1 )
* payment_method (applepay, googlepay)
* payment_network (visa, mastercard, etc)
* payment_token (from apple)
* postal_code
* recurring (1 for yes, 0 or null for no)
* state_province (2 letter code CO)
* street_address (123 Street Drive)
Return
* order_id
* gateway_trasaction_id (not sure if you need this, ADYEN123)
* status (success or error)
* error_message: (will return the message localized in the language passed through)
Example response
Success
{"response":{"status":"success","gateway_transaction_id":"123456TEST82","order_id":"1235.1"}}
Error
{"response":{"status":"error","error_message":"There was an error processing your request. Error reference: 1234.5","gateway_transaction_id":null,"order_id":”1234.5”}}
==== Apple Pay Testing Links (TBC) ====
* Apple Pay Developer Account login
* Apple Pay test cards
* Fundraising-tech Apple Pay Sandbox tester accounts 
* Fundraising-tech Apple Pay Test Server List
==== Verifying Merchant Domains (payments.wikimedia.org) ====
* Navigate to the Merchant Identifiers list here
* Select the 'Wikimedia Apple Pay Merchant ID' record to edit
* On the Edit or Configure Merchant ID page, Scroll down to the Merchant Domains block
* On the right, click Verify
* Download the new verification file payments.wikimedia.org
* In payments-wiki, update the contents of .well-known/apple-developer-merchantid-domain-association.txt with the contents of the new verification file you just downloaded
* Deploy that change
* Reverify by clicking 'OK'
It will take a few seconds to process, and then if successful, you will be redirected back to the Edit or Configure Merchant ID page where Verification Expires date should be updated.
=== Google Pay ===
Spike: https://phabricator.wikimedia.org/T244513
Sandbox test:
* Use https://groups.google.com/forum/#!forum/googlepay-test-mode-stub-data to join test group, then you can test google pay with test card implemented by google already
* Currently support payment methods are inside the `/src/payments/extensions/DonationInterface/adyen_gateway/checkout/adyen_checkout.adapter.php` as getGoogleAllowedNetwork, includes 'AMEX', 'DISCOVER', 'JCB', 'MASTERCARD', 'VISA'
== Recurring == 
Recurring Adyen donations are tokenized and are charged on our schedule by the SmashPig Civi extension's recurring payment charge job. Adyen recurring donations require two identifiers:
* Standard payment token, stored in the civicrm_payment_token table and called recurring_payment_token in queue messages and SmashPig parameters, referred to as 'storedPaymentMethodId' in the Adyen API
* Shopper identifier, stored in civicrm_contribution_recur as invoice_id and called processor_contact_id in queue messages and SmashPig parameters, referred to as shopperReference in the Adyen API
== Auto Rescue == 
Project phab: https://phabricator.wikimedia.org/T340159
https://docs.adyen.com/online-payments/auto-rescue/cards/
=== Config ===
There are two config settings, enabling and the amount of days to rescue for, defaults in Smashpig adyen's main.yaml
enable-auto-rescue: true
max-days-to-rescue: 16
== Subscription Models ==
We have three different situations of recurring/non recurring models in the Adyen console
=== One time donation with no monthly convert ===
Shopper interaction - Ecommerce
=== One time donation with monthly convert or the first payment of a recurring ===
Shopper interaction - Ecommerce
Recurring contract RECURRING,ONECLICK
Recurring processing model Subscription
=== Subsequent recurring charges (not the first one) ===
Shopper interaction ContAuth
Recurring contract RECURRING
Recurring processing model Subscription
== IPN messages ==
https://docs.adyen.com/development-resources/webhooks
We continue to receive Adyen IPN messages in SOAP format, as with the HPP integration. Because with Checkout we are able to capture card payments directly from payments-wiki, we have disabled the asynchronous capture flow that we were using with the HPP integration. Code for async capture is still present in SmashPig but is disabled in config/adyen/main.yaml by setting capture-from-ipn-listener: false. We should probably remove that code and config variable as part of T291448.
The IPN messages have an eventCode that we use to identify them and then either do something or ignore the notification.
List of possible eventCodes here: https://docs.adyen.com/development-resources/webhooks/understand-notifications
==== RECURRING_CONTRACT ====
This notification comes in when a new recurring payment is created. Right now we are ignoring all of them except recurring iDEAL.
==== OFFER_CLOSED ====
iDEAL only When a Pending/Received payment is not authorized.
When Adyen returns a Pending or Received result from a redirect: "The shopper has completed the payment but the final result is not yet known. It may take minutes or hours for the iDEAL payments network to confirm this."
The final result of the payment: "You will receive the final result of the payment in an AUTHORISATION notification. If the transaction is not authorised within 30 minutes, the offer will close. Optionally, you can be informed of this through an OFFER_CLOSED notification"
== Audits ==
We are using two different reports for the Adyen audit. The original report used was the weekly file that comes out on friday, this is called the Settlement Detail Report. The nightly file was added afterwards and is called the Payment Accounting Report.
We get an IPN message with the report path and download it using a DownloadReportJob under the jobs-adyen job queue runner.
Files are named settlement_detail_report_batch_1.csv where the number increases and payments_accounting_report_2023_01_1.csv.
We process them with the wmf_audits drupal module that calls functions from SmashPig to parse the CSVs.
=== Adding data to the reports ===
Adyen lets us control what columns are in the reports. To see this configuration you need to be on our company account level in the Adyen console. Once you are on the company account, Settings->Report columns should appear on the left menu.
=== File scheduling ===
The schedules for the report creation are on the company account level under Reports on the left menu. There are also reports available on the merchant level account but they are not all there. Each report can then be set to Automatic generation, some have specific schedules you can set and some are determined by Adyen.
== Pending Table ==
The DonationInterface Adyen Checkout adapter as of November 2021 sends duplicated messages to the Pending queue. It might not need to send any messages for card payments, as that queue is mostly used for pending transaction resolution, and that scenario is much less likely under Adyen Checkout due to us making the payment authorization and capture in the same HTTP request where we get the donor data.
== Jobs Queue ==
The IPN listener sends 'RecordCaptureJob' messages to the Jobs-Adyen queue. These jobs combine information from the IPN message with information from the pending queue and send information to the donations queue.
There is also RecurringContractJob which are used for creating new iDEAL recurrings. They combine the recurring token from the IPN message from the original donation information in the pending queue and send the combined information to the donations queue.
For overall payment queue can go to server, then run `drush --user=1 -v -r /srv/civi-sites/wmff/drupal qc` to consume the queues
== Testing ==
https://www.mediawiki.org/wiki/Fundraising_tech/Testing#Adyen