Package: meterstick
Meterstick DocumentationThe meterstick package provides a concise syntax to describe and execute
routine data analysis tasks. Please seemeterstick_demo.ipynbfor examples.DisclaimerThis is not an officially supported Google product.tl;drModify the demo colabnotebookand adapt it to your needs.Building up an analysisEvery analysis starts with aMetricor aMetricList. A full list of Metrics
can be found below.AMetricmay be modified by one or moreOperations. For example, we might
want to calculate a confidence interval for the metric, a treatment-control
comparison, or both.Once we have specified the analysis, we pass in the data to compute the
analysis on, as well as variables to slice by.Here is an example of a full analysis:# define metricscvr=Ratio("Conversions","Visits")bounce_rate=Ratio("Bounces","Visits")(MetricList((cvr,bounce_rate))|PercentChange("Experiment","Control")|Jackknife("Cookie",confidence=.95)|compute_on(data,["Country","Device"]))This calculates the percent change in conversion rate and bounce rate,
relative to the control arm, for each country and device, together with
95% confidence intervals based on jackknife standard errors.Building Blocks of an Analysis ObjectMetricsA Meterstick analysis begins with one or more metrics.Currently built-in metrics include:Count(variable): calculates the number of (non-null) entries ofvariableSum(variable): calculates the sum ofvariableMean(variable): calculates the mean ofvariableMax(variable): calculates the max ofvariableMin(variable): calculates the min ofvariableRatio(numerator, denominator): calculatesSum(numerator) / Sum(denominator).Quantile(variable, quantile(s)): calculates thequantile(s)quantile forvariable.Variance(variable, unbiased=True): calculates the variance ofvariable;unbiaseddetermines whether the unbiased (sample) or population estimate
is used.StandardDeviation(variable, unbiased=True): calculates the standard
deviations ofvariable;unbiaseddetermines whether the unbiased or MLE
estimate is used.CV(variable, unbiased=True): calculates the coefficient of variation ofvariable;unbiaseddetermines whether the unbiased or MLE estimate of
the standard deviation is used.Correlation(variable1, variable2): calculates the Pearson correlation
betweenvariable1andvariable2.Cov(variable1, variable2): calculates the covariance betweenvariable1andvariable2.All metrics have an optionalnameargument which determines the column name
in the output. If not specified, a default name will be provided. For instance,
the metricSum("Clicks")will have the default namesum(Clicks).Metrics such asMeanandQuantilehave an optionalweightargument that
specifies a weighting column. The resulting metric is a weighted mean or
weighted quantile.To calculate multiple metrics at once, create aMetricListof the individualMetrics. For example, to calculate both total visits and conversion rate,
we would write:sum_visits=Sum("Visits")MetricList([sum_visits,Sum("Conversions")/sum_visits])When computing analyses involving multiple metrics, Meterstick will try to
cache redundant computations. For example, both metrics above require
calculatingSum("Visits"); Meterstick will only calculate this once.You can also define custom metrics. See sectionCustom Metricbelow for
instructions.Composite MetricsMetrics are alsocomposable. For example, you can:Add metrics:Sum("X") + Sum("Y")orSum("X") + 1.Subtract metrics:Sum("X") - Sum("Y")orSum("X") - 1.Multiply metrics:Sum("X") * Sum("Y")or100 * Sum("X").Divide metrics:Sum("X") / Sum("Y")orSum("X") / 2.
(Note that the first is equivalent toRatio("X", "Y").)Raise metrics to a power:Sum("X") ** 2or2 ** Sum("X")orSum("X") ** Sum("Y")....or any combination of these:100 * (Sum("X") / Sum("Y") - 1).Common metrics can be implemented as follows:Click-through rate:Ratio('Clicks', 'Impressions', 'CTR')Conversion rate:Ratio('Conversions', 'Visits', 'CvR')Bounce rate:Ratio('Bounce', 'Visits', 'BounceRate')Cost per click (CPC):Ratio('Cost', 'Clicks', 'CPC')OperationsOperations are defined on top of metrics. Operations include comparisons,
standard errors, and distributions.ComparisonsAcomparisonoperation calculates the change in a metric between various
conditions and a baseline. In A/B testing, the "condition" is
typically a treatment and the "baseline" a control.Built-in comparisons include:PercentChange(condition_column, baseline): Computes the percent change
(other - baseline) / baseline.AbsoluteChange(condition_column, baseline): Computes the absolute change
(other - baseline).MH(condition_column, baseline, stratified_by): Computes theMantel-Haenszel estimator.
The metric being computed must be aRatioor aMetricListofRatios.
Thestratified_byargument specifies the strata over which the MH
estimator is computed.CUPED(condition_column, baseline, covariates, stratified_by): Computes
the absolute change that has been adjusted using theCUPEDapproach. See thedemofor details.PrePostChange(condition_column, baseline, covariates, stratified_by):
Computes the percent change that has been adjusted using thePrePostapproach. See thedemofor details.Example Usage:... | PercentChange("Experiment", "Control")Note thatcondition_columncan be a list of columns, in which casebaselineshould be a tuple of baselines, one for each condition variable.Standard ErrorsAstandard erroroperation adds the standard error of the metric
(or confidence interval) to the point estimate.Built-in standard errors include:Jackknife(unit, confidence): Computes a leave-one-out jackknife estimate
of the standard error of the child Metric.unitis a string for the variable whose unique values will be resampled.confidencein (0,1) represents the level of the confidence interval;
optionalBootstrap(unit, n_replicates, confidence): Computes a bootstrap estimate
of the standard error.n_replicatesis the number of bootstrap replicates, default is 10000.unitis a string for the variable whose unique values will be resampled;
ifunitis not supplied the rows will be the unit.confidencein (0,1) represents the level of the confidence interval;
optionalPoissonBootstrap(unit, n_replicates, confidence): Computes a Poisson
bootstrap estimate of the standard error. It's identical toBootstrapexcept that we usePoisson(1)instead of multinomial distribution in
sampling. It's faster thanBootstrapon large data when computing in SQL.
See thepostonThe Unofficial Google Data Science Blogfor a good introduction.Example Usage:... | Jackknife('CookieBucket', confidence=.95)DistributionsAdistributionoperation produces the distribution of the metric over
a variable.Distribution(over): calculates the distribution of the metric over the
variables inover; the values are normalized so that they sum to 1. It has
an aliasNormalize.CumulativeDistribution(over, order=None, ascending=True, sort_by_values=False):
calculates the cumulative distribution of the metric over the variables inover. Before computing the cumulative sum, we sort by the values ifsort_by_values=Trueelse by theovercolumn(s). Ifsort_by_values=False,
you can pass in a list of values as a customorder.ascendingdetermines
the direction of the sort.Example Usage:Sum("Queries") | Distribution("Device")calculates the
proportion of queries that come from each device.DiversityAdiversityoperation measures how diverse the child metric values are.HHI(over): calculates the Herfindahl–Hirschman index of the metric values
over the variables inover. The metric values are first normalized overoverthen the HHI is computed.Entropy(over): calculates the entropy of the metric values
over the variables inover. The metric values are first normalized overoverthen the entropy is computed.TopK(over, k): calculates the total share of the top k contributors. The
metric values are first normalized overoverthen largest k values are
summed.Nxx(over, x): calculates the minimum number of contributors to achievextotal share. The metric values are first normalized overoverthen we
count the largestncontributors that make upxtotal share.ModelsA MeterstickModelfits a model on data computed by children Metrics.Model(y, x, groupby).compute_on(data)is equivalent toComputesy.compute_on(data, groupby)andx.compute_on(data, groupby).Fits the underlying model on the results from #1.We have built-in support forLinearRegression,Ridge,Lasso,ElasticNetandLogisticRegression. Example Usage:LinearRegression(Sum('Y'), Sum('X'), 'country')calculates the sum of Y and X by country respectively, then fits a
linear regression between them.Note thatx, the 2nd arg, can be a Metric, a MetricList, or a list of Metrics.FilteringWe can restrict our metrics to subsets of the data. For instance to calculate
metrics for non-spam clicks you can add awhereclause to the Metric or
MetricList. This clause is a boolean expression which can be passed to pandas'query() method.sum_non_spam_clicks=Sum("Clicks",where="~IsSpam")MetricList([Sum("Clicks"),Sum("Conversions")],where="~IsSpam")Data and SlicingOnce we have specified the metric(s) and operation(s), it is time to
compute the analysis on some data. The final step is to pass in the data,
along with any variables we want to slice by. The analysis will be carried out
for each slice separately.The data can be supplied in two forms:a pandasDataFramea string representing a SQL table or subquery.Example Usage:compute_on(df, ["Country", "Device"])Example Usage:compute_on_sql("SELECT * FROM table WHERE date = '20200101'", "Country")Customizing the Output FormatWhen calculating multiple metrics, Meterstick will store each metric as a
separate column by default. However, it is sometimes more convenient to store
the data in a different shape: with one column storing the metric values and
another column storing the metric names. This makes it easier to facet by metric
in packages likeggplot2andaltair. This is known as the "melted"
representation of the data. To return the output in melted form, simply add the
argumentmelted=Truein compute_on() or compute_on_sql().VisualizationIf the last operation applied to the metric isJackknifeorBootstrapwith
confidence, the output can be displayed in a way that highlights significant changes by calling.display().You can customize thedisplay. It takes the same arguments as the underlying
visualizationlibrary.You can visualize the Metric tree by callingvisualize_metric_tree(rendering_fn), whererendering_fnis a function that
can render a string ofDOT representation.
It can help you to sanity check complex Metrics.SQLYou can get the SQL query for all built-in Metrics and Operations (except
weighted Quantile) by callingto_sql(sql_data_source, split_by)on the Metric.sql_data_sourcecould be a table or a subquery. The
dialect it uses is thestandard SQLin Google Cloud's BigQuery. For example,MetricList((Sum('X',where='Y > 0'),Sum('X'))).to_sql('table','grp')givesSELECTgrp,SUM(IF(Y>0,X,NULL))ASsum_X,SUM(X)ASsum_X_1FROMtableGROUPBYgrpVery often what you need is the execution of the SQL query, then you can callcompute_on_sql(sql_data_source,split_by=None,execute=None,melted=False,mode=None)directly, which will give you a output similar tocompute_on().executeis a
function that can execute SQL query. Themodecan beNoneor'mixed'. The former is recommended and computes things in SQL whenever
possible while the latter only computes the leaf Metrics in SQL.Apache BeamThere is also acompute_on_beam(pcol,split_by=None,execute=None,melted=False,mode=None)method which takes anPCollectionwith aschemaas input. The args are similar to those ofcompute_on_sqlexcept thatexecutenow should evaluate aPCollection.
Under the hood, we generate SQL queries and pass them toSqlTransform.
As a result,You need to choose a Beam runner that supportsSqlTransform. For example,
theInteractiveRunnerdoesNOT.The config of the pipeline that carries thePCollectionis set up by you.
For example, your setup decides if the pipeline will be ran in process or in Cloud.Custom MetricYou can write your own Metric and Operation. Below is a Metric taken from the democolab.
The Metric fits a LOWESS model.importstatsmodels.apiassmlowess=sm.nonparametric.lowessclassLowess(Metric):def__init__(self,x,y,name=None,where=None):self.x=xself.y=yname=nameor'LOWESS(%s~%s)'%(y,x)super(Lowess,self).__init__(name,where=where)defcompute(self,data):lowess_fit=pd.DataFrame(lowess(data[self.y],data[self.x]),columns=[self.x,self.y])returnlowess_fit.drop_duplicates().reset_index(drop=True)As long as the Metric obeys somerules, it
will work with all built-in Metrics and Operations. For example, we can pass it
toJackknifeto get a confidence interval.jk=Lowess('x','y')|Jackknife('cookie',confidence=0.9)|compute_on(df)point_est=jk[('y','Value')]ci_lower=jk[('y','Jackknife CI-lower')]ci_upper=jk[('y','Jackknife CI-upper')]plt.scatter(df.x,df.y)plt.plot(x,point_est,c='g')plt.fill_between(x,ci_lower,ci_upper,color='g',alpha=0.5)plt.show()