Metrics Platform/Conduct an experiment

An experiment is a test of a hypothesis designed to provide trustworthy and generalizable data. It imposes an intervention on subjects with the intention of observing what outcome that intervention leads to.mw:Product_Analytics/Glossary#Experiment This page describes how to create an experiment using the Metrics Platform.
== Plan ==
=== Experimentation scorecard ===
[In beta] The experimentation scorecard (restricted access while in beta) provides a template for creating an experiment.
=== Measurement plan ===
=== Instrumentation spec ===
 For more information about designing an instrumentation spec, see the instrument guide.
=== Data collection guidelines ===
== Code ==
You can write your experiment code in the WikimediaEvents extension on in your product codebase. See API Docs to see how Metrics Platform can help you and this Example if you want to see how to use a standard instrument to run a simple A/B test in MediaWiki.
=== Experiment enrollment sampling ===
Experiment enrolment sampling is the act of enrolling users into experiments and consistently assigning an enrolled user a variant of the feature that is being experimented on.This section is based on the discussion in T372108 Document desired properties of an enrolment sampling algorithm
An experiment enrolment sampling algorithm, therefore, is a function, method, or process that accepts some inputs and returns a variant, i.e.module Experiments {
 enroll( user: User, experiment: Experiment ): Variant;
}
Where:
* user is a token that represents that user for at least the duration of the experiment; and
* experiment is one or more constants that define the experiment, e.g. name, sample rate, and variants
==== Properties ====
Such an algorithm must:
* Ensure consistency of assignment within experiment. For example, if there are two experiments running, both having two variants, then the same user should be assigned the same variant for the same experiment in such a way so as to ensure that the following assignments are equally likely:
Experiment 1	Experiment 2
Variant 1 	Variant 1
Variant 1 	Variant 2
Variant 2 	Variant 1
Variant 2 	Variant 2
and not:
Experiment 1	Experiment 2
Variant 1 	Variant 1
Variant 2 	Variant 2
* Be able to sample on a variety of levels, e.g. page, session, user, application install
* Sample when needed, e.g. sample when a user visits a specific page and thereby not assign groups to users who never visit that page
* Not require a backing store
==== Caveats ====
In order for the first and last properties mentioned above to hold, any system using such an algorithm must lock or freeze the inputs to the algorithm for the duration of the experiment. However, it should be OK to extend the end date of an in-progress experiment.
== Launch ==
Once your experiment code has been deployed to production, complete the launch-your-experiment form to configure it and start collecting data. Note that, once the experiment is registered, it won't start collecting data until you turn it on.
== Monitor ==
You can use the experimentation directory to monitor the progress of your experiment. Your event data will be available in a Hive table with the same name as the event stream, with dots and dashes replaced with underscores.
Keep in mind that, for now, this monitoring feature is not done yet in MPIC and there is no a way to monitor the progress of your experiment using that tool. Anyway, you can review your data in Hive and create a superset dashboard to monitor how your experiment is working.
== Manage ==
The experimentation directory provides options to edit and turn off your experiment using the Actions menu.
Keep in mind that, once your experiment has been activated for the first time, you won't be able to edit the following details (even if you turn it off before editing): Name, Sampling and Variants
=== Decommissioning ===
Instruments that capture metrics related to an experiment should be disabled once the experiment is complete. To decommission an experiment:
# Turn off the experiment using the Actions menu.
# Remove the instrument code that calls a Metrics Platform client API.
== Example ==
This section shows how one can use the standard CTR instrument to run a simple A/B test in MediaWiki.
Note that in the case of MediaWiki, the WikimediaEvents extension contains modules that can be used for adding instrumentation and experimentation code.
Once you've gone through the preliminary steps of designing your experiment, you can use EPIC (fka MPIC) to manage your experiment configuration. Assuming that the experiment is named Experimentation Lab Test 1 Experiment with a feature name called experimentation_lab_test_1_feature, and that it is using the web base schema, this experiment would need to be activated once you are ready to start the experiment.
The following javascript file (with matching references to the corresponding experiment slug in EPIC/MPIC) would be named ExLabTest1.js and included in the /ext.wikimediaEvents.exLab directory (note that it is using the main menu hide button in the Vector skin as the selector for the CTR instrument): 
const userExperiments = mw.config.get( 'wgMetricsPlatformUserExperiments' );
const selector = '[data-pinnable-element-id="vector-main-menu"] .vector-pinnable-header-unpin-button';
const friendlyName = 'pinnable-header.vector-main-menu.unpin';
const experimentName = 'experimentation-lab-test-1-experiment';
const featureName = 'experimentation_lab_test_1_feature';
/**
 * Experimentation Lab's first test module
 *
 * This module includes the ClickThroughRateInstrument for testing A/B test enrollment.
 *
 * Note this is temporary code - it will be removed once we validate data collection (T383801).
 */
const ExLabTest1 = {
	init() {
		if ( !mw.eventLog.isCurrentUserEnrolled( experimentName ) ||
			!( featureName in userExperiments.assigned ) ) {
			return;
		}
		const ClickThroughRateInstrument = require( './ClickThroughRateInstrument.js' );
		if ( userExperiments.assigned[ featureName ] === 'true' ) {
			ClickThroughRateInstrument.start( selector, friendlyName );
		}
	}
};
module.exports = ExLabTest1;
The above code would need to be initialized in /ext.wikimediaEvents.exLab's index.js file:
module.exports = {
	ClickThroughRateInstrument: require( './ClickThroughRateInstrument.js' )
};
// ---
// Experimentation Lab Test 1
// ==========================
// This part of the file initializes the first end-to-end test of the Experimentation Lab (ExLab).
//
// Note that this is temporary code - it will be removed once Experiment Platform validate data
// collection (T383801).
const exLabTest1Enabled = require( './config.json' ).exLabTest1Enabled;
if ( exLabTest1Enabled ) {
	const ExperimentationLabTest1 = require( './ExLabTest1.js' );
	mw.requestIdleCallback( () => {
		ExperimentationLabTest1.init();
	} );
}
The following configuration would be needed in the wiki in which the experiment is being run (in this case testwiki):
'wgMetricsPlatformEnableExperiments' => [
	'default' => false,
	'testwiki' => true,
],
'wgMetricsPlatformEnableStreamConfigsFetching' => [
	'default' => false,
	'testwiki' => true,
],
'wgMetricsPlatformEnableStreamConfigsMerging' => [
	'default' => false,
	'testwiki' => true,
],
// Enables a test module for verifying experiment enrollment.
// This config var is temporary - it will be removed in T383801
'wgWMEExLabTest1Enabled' => [
	'default' => false,
	'testwiki' => true,
],
The name of the experiment javascript file would need to be included in the ResourceLoader module /ext.wikimediaEvents.exLab of WME's extension.json file (note that you need to add a config var for that will be used to add the experimentation lab module):
"ext.wikimediaEvents.exLab": {
	"localBasePath": "modules/ext.wikimediaEvents.exLab",
	"remoteExtPath": "WikimediaEvents/modules/ext.wikimediaEvents.exLab",
	"packageFiles": [
		"index.js",
		"ClickThroughRateInstrument.js",
		"ExLabTest1.js",
		{
			"name": "config.json",
			"config": {
				"exLabTest1Enabled": "WMEExLabTest1Enabled"
			}
		}
	],
	"dependencies": [
		"ext.eventLogging"
	]
}
You will also need to ensure the WME hook (WikimediaEventsHooks.php) is updated with the config variable conditional check for loading the module - in this case it is using the BeforePageDisplay hook:
if ( ExtensionRegistry::getInstance()->isLoaded( 'MetricsPlatform' ) &&
	$this->config->get( 'WMEExLabTest1Enabled' ) ) {
		$out->addModules( 'ext.wikimediaEvents.exLab' );
}
Once the experiment code and configuration are in production and the experiment is launched in EPIC/MPIC, you should start seeing events populate the corresponding table in Hive (if using the web base stream, the table to query is product_metrics_web_base).
== References ==