nyuuzyou/wikis huggingface dataset:
Fundraising/Internal-facing/CiviCRM/Exporting SearchKit Search into an extension

This is necessary, if the extension is designed to be reused and the search is an integral aspect of the usability of the form. The following steps would describe how to ensure a SearchKit search is packaged into a custom extension.
# On the top left Navigation bar, select Search and in the dropdown, click on SearchKit.
# In the custom searches, filter the list by entering the name of the search created in the steps above.
# On the row that corresponds to the target search, select the hamburger menu on the extreme right and click on Export.
# In the popup dialog, click on the API Explorer Export link in the highlighted text. This would cause the page to navigate to the CiviCRM APIv4 page preconfigured to the export query on the SavedSearch entity.
# Click on the Execute button to initiate the query.
# In the Result tab at the bottom of the page, set the result formatter to "View as PHP" as the next step would be dependent on that.
# Click on the "Copy" button to copy the entire result.
# On the extension directory in the file system, create a new directory titled "managed"
# In this "managed" directory create a new php file with title {desired_name}.mgd.php where the variable desired_name could be any name you choose. The most important aspect of this is the file extension that is ".mgd.php".
# In this new file, initiate the php script with the php opening tag "<?php"
# Then paste in the php code copied from the API Explorer Export results in step 7.
# Save the file and now the SearchKit search is re-usable anywhere the extension is installed.
# Whenever the extension is installed, the SearchKit search would be found in the "Packaged Searches" tab
====== Related Topics: ======
* Exporting FormBuilder Pages into an extension
* Create Custom Formbuilder pages to monitor tables in other WMFF Databases
* Creating extensions in Civicrm to manage tables in other WMFF Databases
Fundraising/Internal-facing/CiviCRM/Triggers

= Overview =
We use triggers to store forensic information using CiviCRM's core logging functionality - we also alter these in
wmf_civicrm_civicrm_triggerInfo in wmf_civicrm.module
===== Update triggers on production =====
We use mysql triggers to log civicrm database updates to the log tables. These triggers are managed by CiviCRM. However, our production user does not have enough mysql permissions to create the triggers within mysql. To get around this we use a CiviCRM setting on production to log the sql to update the triggers to a file rather than live update them. We then check this file into our crm repo (sites/all/modules/wmf_civicrm/scripts/triggers.mysql currently) and fr-Ops run the file on live.
On development environments triggers are automatically updated in the database - which is generally easier - but to make your local output the triggers as live does the logging_no_trigger_permission setting can be enabled
drush @wmff cvapi Setting.create logging_no_trigger_permission=1
Trigger generation needs to be done on production as the fields differ slightly on staging / dev environments. There are a few methods but turning logging off & back on generally generates consistent output. ie
 drush @wmff cvapi Setting.create logging=0
 drush @wmff cvapi Setting.create logging=1
This will generate a file named something like CiviCRM.trigger62451ae5ab5a5mYm67702126718965e4a41105a08d6202e60.sql that will be in drupal/sites/default/files/civicrm/ConfigAndLog/ - copy this back to your home drive and scp it back to your local machine as sites/all/modules/wmf_civicrm/scripts/triggers.mysql. Changes to this files are committed, reviewed and deployed but they will not be 'live' until fr-tech-ops loads them so once deployed they need to be engages to run the latest triggers.mysql file
Fundraising/Internal Endpoints/CiviCRM extensions

CiviCRM extensions (checked in to drupal/sites/default/civicrm/extensions/ in the CRM repo):
* Extended Mailing Stats Moslty adds fields for storing extra mailing statistics (opens etc)
* Email Amender Corrects common errors in domain names in email addresses
* Deduper Combines multiple contact records for the same person
* Matching Gifts Retrieves lists of employer matching gift policies with a third party provider. Also exports the list of employers and IDs so that we can use it in front-end forms
* Monolog Use monolog streams to output logging to e.g. syslog rather than the files written by Civi core.
* Export Permission Allows us to disable exporting search results for certain user roles
* CiviToken Provides extra tokens for anything Civi renders from templates
* Extended Report More report templates
* Angular Profiles - this was required for Contact layout editor - it is likely we no longer need it
* Contact Layout Editor GUI for editing contact summary screen
* Shoreditch theme based on BootstrapExtended Mailing Stats
* Email Amender Corrects common errors in domain names in email addresses
* Deduper Combines multiple contact records for the same person
* Matching Gifts Retrieves
* Tutorial
* Contact Editor Allows changing contact types
* Data Checks Detects and batch-fixes contact data inconsistencies such as no primary location.
* Forgetme UI for complying with privacy data deletion requests
* Wikimedia Geocoder Privacy respecting but imprecise address geocoder that uses a local zip code database
* Omnimail Integrates with bulk email providers (Acoustic is most fully implemented). Attaches CiviCRM activities to contacts based on email sends from bulk mail providers
* Relationship Block Shows relationships in contact summary
* RIP Sets the "No Bulk E-mail (User opt out)" flag on deceased people
* SmashPig Uses the SmashPig library to make nightly recurring charges for monthly donors
* Wikimedia System Tools debugging helpers
* The The Normalizes use of articles in organization names
* TargetSmart Helper for adding contact data from TargetSmart
* Unsubscribe Email Dedicated screen to facilitate opting contacts out by email address
* WMF Fraud reports Helps the payments team review suspicious transactions when deciding whether to capture, reject, or refund.
* Replay-On-Write Allows using CiviCRM with a Primary+Replica database setup
* Activity Assignee Settings Limit who can be assigned activities
* WMF CiviCRM Catchall for our site-specific code which has migrated out of Drupal modules
* WMF Thank You Extension for handling our automatically donors thank yous, monthly converts and end of year messages
Drupal modules (Checked in at drupal/sites/all/modules/ )
See outdated list at https://wikitech.wikimedia.org/wiki/Fundraising#Drupal_modules
We use the civix utility to generate and update Civi extension boilerplate.
Fundraising/Queue wrangling

When a lot of adyen or amazon messages show up in the damaged queue with MISSING_PREDECESSOR, it's usually because they can't find matching pending queue messages. Here's how to recreate those from logs:
It was possible, but a bit tedious. Need to script out much of this:
php Maintenance/EmptyQueueToDump.php --no-ack --queue donations-damaged --outfile ~/damagedDonations.json -m 100 --config-file ~/SmashPigConfig.php --raw
grep -o 'completion_message_id."[^"]*"[^"]\+' ~/damagedDonations.json | sed -e 's/.*astropay-/astropay-/' -e 's/.$//' > ~/missingPendingMessages
(move missingPendingMessage to log box)
zgrep -F -f ~/missingPendingMessages payments-astropay-20160[45]* > ~/missingAstroPayLogJson
then, using this raw message template as apcontainer:
raw={"headers":{"expires":"0","source_host":"something","correlation-id":"CORRID","source_enqueued_time":"1458060070","destination":"\/queue\/pending","php-message-class":"SmashPig\\CrmLink\\Messages\\DonationInterfaceMessage","source_type":"payments","priority":"4","source_version":"unknown","source_run_id":"2027","persistent":"true","source_name":"DonationInterface","gateway":"astropay","timestamp":"1458060070309"},"body":"
create a file that we can shove into the pending queue:
for corrid in `cat missingPendingMessages`
do
body=`grep $corrid missingAstroPayLogJson | sed -e 's/\\/\\\\/g' -e 's/"/\\"/g'`
echo `sed -e "s/CORRID/$corrid/" apcontainer`$escbody\"\} >> syntheticMessages.json
done
(move syntheticMessages to smashpig box)
php Maintenance/PopulateQueueFromDump.php --config-file config.php --queue pending ~/syntheticMessages.json
Fundraising/RFC/Abstract CRM import

This is a proposed extension to CiviCRM, which will facilitate decoupled payment processing and event registration, nightly auditing of donations, and importing from spreadsheet.
The basic framework is that it normalizes inputs and performs various configurable tests on the data quality. New or updated contacts, participations, contributions, and related entities are encoded by a source-specific plugin and then imported in a generalized way.
TODO:
* reconcile with existing work
* plan multi-entity import -- importing a contact and a contribution at the same time is not a standard part of the Civi workflow.
== Intake modules ==
=== Messaging ===
A one-directional message-passing protocol in which messages containing an aggregation of entities are sent to CiviCRM by third-parties.
Only a status is returned: was the import successful? We will not pass counts and rejected subentities back across the component boundary—see the transactional model below for how to interpret a success or failure status.
=== CSV ===
=== Auditing ===
== Exception handling ==
The default will be to wrap each item in a transaction. If some of the message fails, all database changes will be rolled back. Likewise, reads will be made against a snapshot of the database at the beginning of intake.
== Metadata ==
The import processes will stamp various diagnostic and contextual values into the message.
; Source metadata : source channel, archived raw message URL
; Intake plugin identity: timestamp at intake, calculated content-type, source revision
; Import job identity: CiviCRM source revision, hostname, contact.created_time
== Fundraising deployment ==
Introduction of this work into the donations pipeline can be phased.
* Phase 0: write the abstract import library. Not really a deployment phase, but a prerequisite.
* Phase 1: upgrade to an abstract import-enabled version of CiviCRM, without changing message intake configuration
* Phase 2: write an intake plugin for a single source. This should be considered a port of the existing import mechanism for that source, with strict feature parity and tests to prove it. Slow-start this plugin, hand-feeding it small batches. Can run in parallel with legacy import. Discuss upstreaming strategy using example code.
* Phase 3: continue writing intake plugins until finished.
* Phase 4: deprecate legacy intake. Reevaluate coarse architecture, e.g. Drupal vs WordPress.
== Data retention ==
Raw intake data blobs are preserved until there is verification of the imported data's integrity (and possibly longer in long-term storage, for tax or legal reasons?).
== See also ==
* https://github.com/eileenmcnaughton/nz.co.fuzion.csvimport, https://civicrm.org/extensions/api-csv-import-gui
* https://github.com/eileenmcnaughton/civimigrate
* http://book.civicrm.org/user/current/common-workflows/importing-data/
Fundraising/RFC/Direct Mail Appeal campaign management

This is a specification for a new feature to allow Fundraising Major Gifts to set up their own Direct Mail appeals.
== Background ==
Several times a year, Major Gifts wants to publish a donate link which causes all donations to be associated with a specific campaign. Currently, the process is to ask FR production to:
# Create a special form with a "utm_campaign=" URL parameter
# Create a CiviCRM Direct Mail Appeal value for the campaign
# Ask FR tech to connect the url parameter with the appeal value, for example
This feature is meant to eliminate the third step, and create a framework which we could build on in the future to potentially make the entire process happen with one form submission.
The FR tech step is to add a case to this switch statement.
== Basic requirements ==
* Major Gifts is able to link the utm_campaign and Direct Mail Appeal using a self-serve form, which persists the associations in a db table.
* View which shows existing campaigns.
* Donate URL including the utm_campaign parameter is generated from a configurable format string and displayed in the campaign editor.
* Please design so that it would be possible to have multiple utm_campaign parameters associated with one DMA campaign.
I'm open to either a WMF-specific implementation done in wmf_civicrm.module, or a generalized CiviCRM extension which might be useful to others.
== Additional development ==
This stuff is optional and should be negotiated between GR and MG:
* Related reports, which detail or summarize response to a campaign.
* Manage Direct Mail Appeal values from within the campaign editor?
Fundraising/Roadmap

We want to make our software easier to maintain and develop, and more accessible both to new developers on our team and to other organizations.
DonationInterface still uses jquery for a lot of form manipulation. It would be nice to move to something more modern like VueJS which is used in other new MediaWiki frontend features.
We want to move as much business logic as possible from DonationInterface to SmashPig. The obvious opportunities for this migration are when we rewrite a payment processor integration. The Amazon, Adyen, and Ingenico integrations all have moved a fair amount of business logic out of DonationInterface. The PayPal integration still manages all the business logic in DonationInterface code. We are planning to rewrite the PayPal integration in the first half of 2022. Once PayPal business logic is in SmashPig we can rewrite the PayPal pending transaction resolver and completely remove DonationInterface as a dependency of our CiviCRM install. That will then let us remove the WmfFramework shim from DonationInterface and simplify some code.
SmashPig has some homegrown features that are mostly handled by external libraries in modern applications. We should tear out the homegrown implementations and use a more modern approach to
* logging (replace with monolog)
* monitoring/failmail
* dependency injection (maybe Symfony containers)
We want to make configuration consistent and understandable across our codebases. DonationInterface and SmashPig each have a way to define defaults for configuration values that can be overridden on a per-processor basis. These two approaches should be harmonized. We should also make it easy to test the trickier parts of our configuration such as donation routing to ensure that we get the results we expect.
We need to move any CiviCRM custom code that interfaces with drupal 7 to instead interface with CiviCRM APIs. This will be an opportunity to clean up and modernize code while we are at it. In particular:
* Damaged message display needs batch delete / requeue and better search
* Audit parsers need better recurring handling, less wasteful log searching
* Large donation notifier and exchange rate updater should be totally generic & shared with other Civi users
Our current banner impressions stats collector and pageview counter is built on an inappropriate framework. We have a replacement (FRUEC) but need to switch over to using it and verify that the stats are still correct.
Fundraising/SmashPig

= Standalone Installation = 
This is where all the IPN listeners live as well as some maintenance scrips.
Live endpoint: https://payments-listener.wikimedia.org/adyen
=CiviCRM Extension=
We have written an extension in CiviCRM to use the SmashPig library to charge recurring payments. It currently supports credit cards via Ingenico and Adyen, and iDEAL and Apple Pay via Adyen.
The SmashPig extension has a settings menu in the Civi UI:
Administer -> System Settings -> SmashPig Settings
Here you can switch the extension to either use queues or directly insert the donation. In production we always want to use queues so that the charge job doesn't conflict with the donation queue consumer for table access, and so that any special data munging we apply in the queue consumer is applied consistently.
The extension defines a CiviCRM PaymentProcessor implementation (limited to only charging recurring installments) and a job that can be run via the CiviCRM.
Running the recurring job with 1 donation 
drush @wmff -vv cvapi job.process_smashpig_recurring batch_size=1
Running with a specific recurring_id 
drush @wmff -vv cvapi job.process_smashpig_recurring contribution_recur_id=12345
Runing through run job (if on prod, always use this instead of above two since we do not want deadlock), this will call the process control script and create a log file
run-job -j recurring_smashpig_charge --slow-start
=Maintenance Scripts=
You can run the SmashPig maintenance scripts from any server with SmashPig installed. Often the CiviCRM server is the most convenient since it's central and can see everything. Some of these are run on a schedule by process-control. All take a --config-node argument which specifies the payment processor config to load (perhaps it should be renamed).
==ConsumePendingQueue==
(Run on schedule)
==CreateIpnMessagesFromPendingDb==
(only for testing, may be obsolete)
==DeleteExpiredPendingMessages==
(Run on schedule)
==DumpConfig==
Print the merged config tree. Handy if you're not sure what's been overridden
php DumpConfig.php --config-node=adyen
==EmptyQueueToDump==
Script to empty out a queue backing store into a file. Objects are not kept in the queue after dumping.
==InitializeSequence==
Reset a sequence generator to a specific number, like the redis counter we use for ContributionTrackingId
==PopulateQueueFromDump==
Counterpart of EmptyQueueToDump.
php Maintenance/PopulateQueueFromDump.php --queue=recurring-modify queueMessage.json
==QueueJobRunner==
(Run on schedule) Runs jobs like those we send to jobs-adyen, jobs-paypal, etc
==QueuePeek==
Display the first message in the queue in print_r format
php Maintenance/QueuePeek.php donations
==RequeueDelayedMessages==
(Run on schedule)
==TestFailMail==
Logs stuff at every level up to alert to trigger a failmail.
Fundraising/Syslog Settings

There are a number of log buckets that we regularly define in syslog, to help us monitor different things happening in DonationInterface, and increasingly, the payments listeners. 
In order to help with debugging on local instances, it is incredibly helpful to set up similar log buckets. 
Our current (July 2013) syslog conf is as follows: 
if ($HOSTNAME startswith 'payments') and ($syslogseverity <= 3) then /var/log/remote/payments.error
&
if ($HOSTNAME startswith 'payments') and ($programname contains '_payment_in') then /var/log/remote/payments-initial
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains '_fraud') then /var/log/remote/payments-fraud
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'globalcollect_gateway_co') then /var/log/remote/payments-globalcollect_commstats
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'globalcollect_gateway') then /var/log/remote/payments-globalcollect
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'adyen_gateway') then /var/log/remote/payments-adyen
& ~
if ($HOSTNAME startswith 'payments') and ($syslogseverity == 7) then /var/log/remote/payments.debug
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'payflowpro_gateway') then /var/log/remote/payments-payflow
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'apache2') then /var/log/remote/payments-apache2_errors
& ~
if ($HOSTNAME startswith 'payments') and ($programname contains 'amazon_gateway') then /var/log/remote/payments-amazon_gateway
& ~
if ($HOSTNAME startswith 'payments') then /var/log/remote/payments-misc
& ~
Category: Fundraising
Fundraising/Team processes

== How we work ==
Fundraising Tech is a Scrum team. We use Phabricator to manage our backlog and we work in 2-week sprints. 
* Fundraising Tech Phabricator dashboard
* Fundraising Tech Definition of Done
* Fundraising Tech Mid-Sprint Checkup
* Fundraising Tech Sprint Retro
* Designated Response Individuals (aka "Chaos Crew")
* Chaos Crew Fortnightly Handover
* Fundraising Tech How we use Phabricator
== Guides ==
* New integration manual
* Campaign Readiness Checklist
* On-call instructions
== Meta ==
* Training
* Tech Roadmap
== Subpages ==
Fundraising/Team processes/Campaign Readiness Checklist

== Scenario 1: New market or new payment option in a market ==
Adopting an entirely new market with new payment methods and no official prior campaign.
=== Internal test ===
* Smoke test locally / sandbox
* Transaction should get to local test Civi
* Should be visible in processor sandbox
** Test cards
** UI review with fr-online
* Deploy so that we can make a payment vs live processor (the internal test)
** For this deploy, ensure that generic links don’t direct to the experimental processor
* Contact local donor to test on production, ensure that test payment gets to live Civi and live processor 
** UI/UX feedback. Did they get TY email?
=== Prep for 1 hour technical functionality test ===
* Resolve any blocking bugs from local donors/payments, DR or fr-online
=== Prep for Campaign ready ===
* Resolve any blocking bugs from 1 hour test
* If recurring, the transactions look correct in PSP console and/or Civi
* Audits are showing donations from 1 hour test
* DR is happy with tools and reporting
* All payments form settings are set to use this processor/payment option as default
== Scenario 2: Switch of PSP ==
From time to time we may move volume between providers for various  reasons.  In this scenario we have to ensure all banners and emails are pointed to the appropriate place.
=== Prep for 1 hour technical functionality test ===
* Check for and resolve any known bugs 
* Ask for confidence check from local donor  (internal re-test)
* Resolve blocking bugs
=== Prep for campaign ready ===
* Resolve any blocking bugs from 1 hour test
* If recurring, the transactions look correct in PSP console and/or civi
* Audits are showing donations from 1 hour test
* DS is happy with tools and reporting
* All payments form settings are set to use this processor/payment option as default
== Scenario 3:  Repeat Campaign ==
We’ve run a campaign in this market previously, we are using the same provider  and we are using the same payment types.  Nothing new has been added.
=== Prep for 1 hour technical functionality test ===
* Check for and resolve any known bugs 
=== Prep for campaign ready ===
* Resolve any blocking bugs from 1 hour test
== Yearly Major Campaigns and Code Restrictions ==
Example timeline
Fundraising/Team processes/Chaos crew handover

When leaving the Chaos Crew rotation, the Primary DRI engineer is expected to provide a summary to the team covering the preceding two weeks of Chaos Crew work . This should also help the new Secondary DRI engineer get up to speed with current issues. 
A suggested format for the update:
Chaos in progress:
* Ticket / Context / Suggested next steps
* Ticket / Context / Suggested next steps
* Ticket / Context / Suggested next steps
Chaos completed:
* Ticket / Deployment Status / Stakeholder feedback(if needed)
* Ticket / Deployment Status / Stakeholder feedback(if needed)
* Ticket / Deployment Status / Stakeholder feedback(if needed)
When leaving the Chaos crew, it is recommended that the leaving team members unassign themselves from any assigned tasks so incoming chaos crew members can pick them up.