Package: cohort-analysis
cohort-analysisOutlineA package for calculating cohort metrics from an activity stream and cohorts data table.UsespandasDataFrames to hold the two tables and provides a class,CohortMetricsto:manage the data within these DataFramesfilter to analyse different user cohortscalculate a variety of important activity metrics (such as weekly active users, monthly active users)calculate cohort retention metricsInstallationTo install from the command line viapip, do:pip install cohort-analysisTo upgrade to the latest version viapip, do:pip install cohort-analysis --upgradeTo use viapipenvput the following in your Pipfile:[packages]
cohort-analysis = ">=0.0.1"Getting StartedTo start, simplyimportcohort_analysisThe principal interface for this library is viacohort_analysis.metrics(an instance ofcohort_analysis.cohort_metrics.CohortMetrics).We have usedpandas-flavorto add a.cohortsaccessor to pandas DataFrames; note that the above import statement will apply this to all DataFrames.Preparing the interfaceYou will need to set up theclickstreamandcohortstables for this interface. There are two ways:via methodscohort_analysis.metrics.set_clickstream(df1)cohort_analysis.metrics.set_cohorts(df2)via the DataFrame accessor we set up for conveniencedf1.cohorts.set_as_clickstream(cohort_analysis.metrics)df2.cohorts.set_as_cohorts(cohort_analysis.metrics)Theclickstreamtable should have two columns,user_idandtimestamp(other columns will be ignored). Every row should record a 'click' (an event of some kind), indicating the responsible user's ID and the timestamp at which it occurred. If your DataFrame has these columns under different names (col1,col2in the example below), you can indicate that when setting theclickstreamas follows:cohort_analysis.metrics.set_clickstream(df1,timestamp_col="col1",user_id_col="col2")Thecohortstable has two mandatory columns,user_idandreference_timestampand is expected, though not required, to have additional columns that can be used to define user cohorts. If your DataFrame has these columns under different names (colA,colBin the example below), you can indicate that when setting thecohortsas follows:cohort_analysis.metrics.set_cohorts(df1,reference_timestamp_col="colA",user_id_col="colB")Theuser_idcolumns in the two tables ought to correspond, i.e. IDs inclickstream.user_idshould be present incohorts.user_id(any that are not will be ignored).Direct interface creationAlternatively, a new instance ofcohort_analysis.cohort_metrics.CohortMetricsspecifying the tables to use can be created as follows:metrics=cohort_analysis.CohortMetrics(clickstream=df1,cohorts=df2)The supplied DataFrames should already have the correct column names as detailed in the previous section.Timezone supportThe default behaviour of the interface is to convert/localise all timestamps to UTC.If your data is in a different timezone, you should change theCohortMetricsinstance's timezone as follows:cohort_analysis.metrics.change_timezone("CET")This should be donebeforesetting the clickstream and cohorts table if those contain timezone-naive timestamps that actually belong to a timezone other than UTC.If creating the interface directly, pass atimezoneparameter instead:metrics=cohort_analysis.CohortMetrics(clickstream=df1,cohorts=df2,timezone="CET")Changing the current timeAt initialisation, the current time in the chosen timezone is set as a reference point for calculations. Actions in the clickstream at future times and users with reference timestamps in the future will be filtered out.However, thecurrent_timestampproperty can be changed as follows:metrics.change_current_timestamp("now")Valid inputs are:"now"(changecurrent_timestampproperty to the current time in the chosen timezone)"last"(changecurrent_timestampto the latest entry inclickstream.timestampandcohorts.reference_timestamp)anypandas.Timestamp(will be converted/localised to the instance's timezone)any parseable datetime string (will be set to the instance's timezone)This allows the user to view the data as it would have appeared at points in the past, handle batched data that doesn't include the latest information, set up deterministic tests etc.Inspecting the interfaceInstances ofCohortMetricshave a string representation implemented. If theclickstreamandcohortstables are both set up,print(cohort_analysis.metrics)will display a summary of the two tables derived from the DataFrames' own string representations. This can be used for quick inspection of the data. If one or both tables are not yet set up, the representation will indicate this instead.Basic UsageOnce theCohortMetricsobject has been prepared, it can be used to derive a variety of activity and cohort retention metrics from theclickstreamandcohortstables.Methods will typically return a pandas DataFrame.Filtering by cohortA copy of the two tables, filtered to remove 'clicks' from users not in thecohortstable, or with timestamps before the associated user'sreference_timestamp, may be returned as follows:clickstream,cohorts=cohort_analysis.metrics.filter_by_cohorts()Additional filtering may be applied by passing a dictionary to this method. Here are some examples:Filter by a column in thecohortstable, selecting activity and user data only for (in this example) users who have the value"United Kingdom"in thecountrycolumn of thecohortstable:cohort_dict={"country":"United Kingdom"}clickstream,cohorts=cohort_analysis.metrics.filter_by_cohorts(cohort_dict)Filter by a column in thecohortstable, selecting activity and user data only for users with one of a number of values in a column (in this example,"Brazil"or"Canada"in thecountrycolumn of thecohortstable):cohort_dict={"country":["Brazil","Canada"]}clickstream,cohorts=cohort_analysis.metrics.filter_by_cohorts(cohort_dict)Filter by values in multiple columns of thecohortstable, selecting activity and user data only for users who (in this example) have the value"India"in thecountrycolumn of thecohortstableandthe value"Social Media"in theacquisition_channelcolumn:cohort_dict={"country":"India","acquisition_channel":"Social Media"}clickstream,cohorts=cohort_analysis.metrics.filter_by_cohorts(cohort_dict)Lists of values may also be passed, as in the previous example.
The logic for combining across columns can be changed - 'AND' logic is used by default, but callingcohort_analysis.metrics.change_filter_logic("OR")beforehand will use 'OR' logic instead. 'NOT' logic can also be used to exclude all the values provided viacohort_dict.Filter by thereference_timestampcolumn in thecohortstable, selecting activity and user data only for users with a reference timestamp during a particular time period (in this example, Q2 2020):cohort_dict={"calendar":"2020-Q2"}clickstream,cohorts=cohort_analysis.metrics.filter_by_cohorts(cohort_dict)This kind of filter can be combined with other filters and may have multiple values passed, just as above. Using the key"reference_timestamp"instead of"calendar"will work in the exact same way.
Possible time periods include:years ("2020")quarters ("2020-Q1")months ("2020-05")weeks ("2020-W-SUN-5"- the fifth week concluding on a Sunday in 2020)days ("2020-05-03")This method is used internally by other methods that generate metrics.Cohort Activity MetricsThecohort-analysislibrary supports two different types of activity metric: activity in calendar intervals and activity in rolling intervals.The methods for computing activity incalendar intervalsare as follows:dau_calendar_daywau_calendar_weekmau_calendar_monthqau_calendar_quarteryau_calendar_yearThese methods all optionally accept a cohort filter dictionary via the inputcohort=cohort_dict.Each of these methods returns a DataFrame with a PeriodIndex covering the full timerange of theclickstreamtable'stimestampcolumn and thecohortstable'sreference_timestampcolumn. It has the following columns:cohort_sizen_active?au(dau,wau,mau,qau,yaurespectively)By default, the cohort size is the number of users with reference timestamps prior to the end of each period in the index. This behaviour can be changed to calculate the number of users with reference timestamps prior to thestartof each period in the index by passingcohort_size_at="start"to any of the methods.Methods apart fromdau_calendar_dayandmau_calendar_monthaccept an additional argument, since the other intervals are not uniquely defined (although there are sensible defaults that we implement).wau_calendar_weekaccepts the optional argument,week_commencing. This accepts any day of the week as a value. The default is"Monday".qau_calendar_quarterandyau_calendar_yearboth accept the optional argumentyear_commencing. This accepts any month of the year as a value. The default is"January".The methods for computing activity inrolling windowsare as follows:rolling_waurolling_maurolling_qauThese methods all optionally accept a cohort filter dictionary via the inputcohort=cohort_dict.Each of these methods returns a DataFrame with a DateIndex covering the full timerange of theclickstreamtable'stimestampcolumn and thecohortstable'sreference_timestampcolumn. It has the following columns:cohort_sizen_active?au(wau,mau,qaurespectively)For each date in the index, these metrics are computed for an N day window whose final day is indicated by the index. For WAU, N=7. For MAU, N=28. For QAU, N=91. The latter two values are chosen due to being multiples of 7 (this eliminates weekly cycles in user activity from the metrics).By default, the cohort size is the number of users with reference timestamps prior to the end of each window whose final day is indicated by the index. This behaviour can be changed to calculate the number of users with reference timestamps prior to thestartof each window by passingcohort_size_at="start"to any of the methods.Cohort Retention RatesA cohort retention rate is the active rate of a user cohort during a specified time-interval sincereference_timestamp.  The current implementation ofCohortMetricshas three different type of retentions:fixed-interval,unbounded-intervalandwmqy-interval. The code below shows the usage on acohortsDataFrame withcountryin the column.retention_type='unbounded'retention_cohorts=[{"country":"Canada"},{"country":"Brazil"}]interval_type="week"metrics=cohort_analysis.metricsmetrics.cohort_retention(retention_type,retention_cohorts,interval_type)# returns a DataFrame indexed by week 0 to latest and columned by given cohortsHereretention_cohortsis a list of cohort filtering dictionaries. This allows the end user to easily compare retention metrics for different cohorts.DevelopmentOnce you've cloned the repository and navigated to it, the best way is to use thepipenvvirtual environment:Make sure that you have thepipenvlibrary:pip install pipenv --upgrade.In the top level directory,/cohort-analysis, runpipenv install --dev(installs virtual environment with development tools).Launchpytest-watchin the virtual environment usingpipenv run ptw. Edit code at your leisure; the test suite will run whenever you save your work.Other quality assurance checks can be run locally:usepipenv run coverageto run the tests and ensure sufficient test coverageusepipenv run mypyfor static type-checkingusepipenv run lint-fixto format the codeContributorsPaddy Alton (paddy.alton@apolitical.co)Charlotte Crabb (charlotte.crabb@apolitical.co)Ashia Ogunlade (ashia.ogunlade@apolitical.co)CY Yang (cy.yang@apolitical.co)(with thanks to the Apolitical engineering and data teams for assistance and review)