Package: theseus-growth
TheseusTheseus provides straightforward tools for cohort analysis and general marketing performance analysis. Theseus was created byEric Benjamin SeufertofHeracles.Theseus is an open source library that provides a set of common functions for use in doing analysis related to product growth: building retention profiles, projecting DAU levels, combining cohorts, segmenting cohorts by age, etc. Theseus can be used for marketing budgeting planning, scenario analysis, marketing campaign analysis, revenue projections, and in a media mix model.Theseus is designed to be used for standalone analysis projects as well as in programmatic business intelligence environments.Theseus is provided as open source software under theMITlicense.Note that Theseus is in abetastate; bugs are to be expected.DocumentationThe documentation for Theseus can be found inthis QuantMar thread.InstallationUse the package managerpipto install Theseus.pipinstalltheseus_growthUsageInclude the theseus_growth libraryimporttheseus_growthasthInstantiate a Theseus objectth=th.theseus()Working with Theseus involves using retention profiles to build cohort projections. To get started with analysis, you'll first build a retention profile usingdaysandretentionvalues, where each day value corresponds to a retention value, starting from Day 1 (ie. the day after a user has entered the product). Retention values should be provided as whole numbers (not decimals), eg. 30% retention for some given day would be represented as 30 and not .30.The retention and day values are provided as lists, the lengths of which must match. Theseus uses the index of the values in thedayslist to associate with a value from theretentionlist, so no need to order the lists.Here's an example:x_data=[1,3,7,14,30,60,90,180]y_data=[80,70,55,50,30,22,10,8]facebook=th.create_profile(days=x_data,retention_values=y_data)print(facebook)In this example, Day 1 retention is set to 80, Day 3 retention is set to 70, Day 7 retention is set to 55, etc. Then, these lists are supplied to thecreate_profilefunction to generate a retention profile (in this case, for Facebook, as per the variable name).The curve fit to the retention data is decided by iterating over a number of different function forms to find the one that fits best with the smallest error. The functions tested are:[ 'log', 'exp', 'linear', 'quad', 'weibull', 'power', 'interpolate' ]. A specific function can be forced onto the data by using theformparameter with thecreate_profilefunction; when theformparameter is not set,create_profiledefaults to finding the best fit function.If youprintthefacebookvariable, the output will reveal a number of pieces of information about the retention profile:{'x':[1,3,7,14,30,60,90,180],'y':[80,70,55,50,30,22,10,8],'y_collapsed':[80.0,70.0,55.0,50.0,30.0,22.0,10.0,8.0],'x_collapsed':[1,3,7,14,30,60,90,180],'interpolation_f':<scipy.interpolate.interpolate.interp1dobjectat0x10c6234f8>,'interpolation_s':<scipy.interpolate.fitpack2.InterpolatedUnivariateSplineobjectat0x10c638588>,'params':{'log':array([11.69432981,0.85932489,91.18858849]),'exp':array([6.81055507e+01,4.01937193e-02,1.00786302e+01]),'linear':array([-0.36314103,58.10116222]),'quad':array([4.23356783e-03,-1.09641452e+00,6.94411850e+01]),'weibull':array([136.70664663,0.99893803]),'power':array([88.3002565,0.3123284]),'interpolate':None},'errors':{'log':61.1068291195336,'exp':101.38898207577283,'linear':1412.367783723572,'quad':364.49321231183075,'weibull':12824.82253493541,'power':440.6176923037875},'best_fit':'log','retention_profile':'best_fit','retention_projection':(array([1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180]),[100.0,80.72474913359267,73.46379035522745,68.40395854720839,64.51667685670971,61.35945860038563,58.70096154455828,56.40491084756263,54.384231538266555,52.579888860453295,50.95000745540398,49.46380467615453,48.097988591666066,46.834508880262675,45.65909322498185,44.56026139229167,43.52864131907821,42.55648257398489,41.63730255356898,40.76562419809498,39.936778210530946,39.14675163202042,38.392070317383634,37.669706592412716,36.97700588336105,36.3116278251625,35.67149854953705,35.054771699032194,34.459796319341784,33.88509022316152,33.32931774346239,32.79127103579817,32.269854271186304,31.764070199363758,31.273008668265803,30.79583676761319,30.331790328468763,29.88016656089222,29.440317651599635,29.01164517522338,28.593595198171947,28.185653974577036,27.78734415043222,27.398221405576763,27.017871474283382,26.645907494353807,26.281967642193266,25.92571301762287,25.576825747436132,25.235007281103094,24.899976855723068,24.57147011044914,24.24923783325221,23.933044825140172,23.622668868865617,23.317899790794513,23.01853860601659,22.724396737987817,22.43529530504138,22.15106446700662,21.87154282596005,21.59657687581418,21.326020496044592,21.059734485374676,20.79758613169244,20.539448814872614,20.285201639528808,20.034729095029036,19.787920740381438,19.544670911838594,19.30487845128266,19.06844645364413,18.835282031775264,18.605296097351072,18.378403156503964,18.154521119018952,17.933571120024055,17.715477353206225,17.500166914670615,17.287569656638155,17.07761805024684,16.870247056785416,16.66539400674492,16.462998486125443,16.263002229482055,16.065349019235967,15.86998459081596,15.676856543229135,15.485914254692815,15.297108802987182,15.11039289021575,14.925720771683615,14.743048188626403,14.562332304542082,14.383531644896777,14.206606039992167,14.031516570797578,13.858225517564094,13.686696311050966,13.516893486206541,13.34878263815699,13.18233038036621,13.01750430483952,12.854272944252656,12.692605735895398,12.53247298732623,12.373845843641718,12.216696256270339,12.060996953206299,11.90672141060432,11.753843825661477,11.602339090716569,11.452182768502269,11.30335106848878,11.15582082426181,11.009569471881221,10.86457502916953,10.720816075882823,10.578271734719507,10.436921653124443,10.296745985849213,10.157725378230722,10.019840950153323,9.883074280660807,9.747407393187231,9.612822741376846,9.47930319546505,9.346832029194132,9.215392907238623,9.084969873116748,8.955547337565534,8.827110067358433,8.69964317454533,8.573132106096097,8.447562633929394,8.322920845309952,8.199193133597916,8.076366189334905,7.954426991652241,7.833362799987498,7.713161146095985,7.5938098263449945,7.475296894278529,7.357610653441469,7.240739650452227,7.124672668313735,7.009398719952898,6.8949070419793514,6.781187088654434,6.668228526062208,6.556021226474144,6.444555262900209,6.33382090381852,6.223808608077022,6.114509019960167,6.005912964414506,5.898011442426906,5.790795626549496,5.684256856566179,5.578386635294848,5.473176624520619,5.368618641055164,5.264704652917132,5.16142677562982,5.058777268631218,4.95674853179267,4.8553331020422945,4.754523650089098,4.65431297724453,4.554694012337748,4.455659808721521,4.357203541365507,4.259318504033715,4.161998106543535,4.065235872103301,3.969025434725765,3.873360536714898,3.778235026223541,3.6836428548796363,3.5895780754783857])}You won't ever actually interact directly with a retention profile variable, but you can see that it contains:The original X and Y (thedaysandretentionlists) data provided;A projection (in theretention_projectionvariable);Two_collapsedvariables that contain the average values for each of thedaysandretentionlists (in this example, only one value was provided for each day, so they_collapsedlist is the same as theylist, which was provided);Aparamsdict that contains coefficients for a number of different shape functions;Some other miscellaneous data, like interpolation models;With the Facebook retention profile created, cohort projections can be generated from it. First, the profile can be visualized with theplot_retentionfunction:th.plot_retention(facebook)Which should output a graph that looks like this:Now a DAU projection based on cohorts can be generated -- in the Theseus library, this is called aforward DAU projection. First, we'll create a list of cohorts, meaning a list containing the numbers of new users that joined the product on a daily basis, with each number representing a sequential day.Then, theproject_cohorted_DAUfunction can be used to create a Pandas DataFrame containing the number of DAU present in the product, given the new users that joined via the cohorts, on the basis of thefacebookretention profile. In this example, the function will take 4 inputs:profile: the retention profile to use;periods: the number of periods to project forwardcohorts: a list of new user valuesstart_date: the date at which the cohorts are added and from which the projection is made#cohorts are daily new user values, eg. the number of new users#joining the product on a given daycohorts=[1000,1000,1000,1000,1000]facebook_DAU=th.project_cohorted_DAU(profile=facebook,periods=50,cohorts=cohorts,start_date=1)print(facebook_DAU)The output of this should look like:12345678910...41\cohort_date...11000807734684645613587564543525...285201000807734684645613587564543...2903001000807734684645613587564...29440001000807734684645613587...298500001000807734684645613...303424344454647484950cohort_date12812772732702662622592552522285281277273270266262259255329028528127727327026626225942942902852812772732702662625298294290285281277273270266[5rowsx50columns]This DataFrame table shows how many of the original cohorts are present on any given day; the cohort numbers run down the Y axis and the days run across the X axis.To see this as a total, theDAU_totalfunction can be used:facebook_total=th.DAU_total(facebook_DAU)print(facebook_total)The output of which should look like:12345678910...41\Value...DAU1000808173424913186454859117274863710000...4312424344454647484950ValueDAU424641824119405839993941388838313779[1rowsx50columns]This table represents the total number of DAU present in the product from those five cohorts over the course of a 50-period timeline.Theproject_cohorted_DAUcan be used to project DAU out given some set of cohorts and a retention profile, but it can also be used to generate the number of new users needed to reach a DAU target over a timeline, given some existing set of cohorts.In this example, thecohortslist contains five cohorts of 1000 new users each. If a marketing analyst wanted to know how manyadditionalcohorts, and of what size, would be needed in order to get the user base to 10,000 DAU, then they could useproject_cohorted_DAUto do that by adding two parameters:DAU_targetandDAU_target_timeline.DAU_targetis the targeted number of DAU, andDAU_target_timelineis the number of days over which the additional new users will be added.In action:facebook_DAU=th.project_cohorted_DAU(profile=facebook,periods=50,cohorts=cohorts,DAU_target=10000,DAU_target_timeline=10,start_date=1)print(facebook_DAU)Should produce the following output:12345678910...\cohort_date...11000807734684645613587564543525...201000807734684645613587564543...3001000807734684645613587564...40001000807734684645613587...500001000807734684645613...60000016131302118411031040...70000001757141812901201...80000000185314951361...90000000019341561...100000000002005...41424344454647484950cohort_date12852812772732702662622592552522290285281277273270266262259255329429028528127727327026626225942982942902852812772732702662625303298294290285281277273270266649648948147446746145444844143575495415325245175095024954884818588579570562553545537529522514962461460459558657756956155354510657647636627617608599590581573[10rowsx50columns]This table reveals that the additional DNU needed to get to 10,000 overall DAU within the 10-period timeframe is: 1613, 1757, 1853, 1934, 2005.Note that this approach seeks to minimize the number of total DNU added on any given day within the timeline.To get only the DNU (new users) values from a forward DAU projection, theget_DNUfunction can be used:#get DNU from a DAU projectionfacebook_DNU=th.get_DNU(facebook_DAU)print(facebook_DNU)The output of which should look like:cohort_date1234567\ValueDNU10001.01000.01000.01000.01710.01881.01994.089...41424344454647484950Value...DNU2090.02171.0...0.00.00.00.00.00.00.00.00.00.0[1rowsx51columns]And to reducefacebook_DAUto only the total DAU values over the projection timeline, theDAU_totalfunction can be used again:12345678910...41\DAU...010001807254132253870509663227548877410000...4384424344454647484950DAU0431842504188412640674009395338973842[1rowsx50columns]Note that this shows DAU reaching 10,000 by Day 10.The Facebookforward DAU projectioncan be visualized with theplot_forward_DAU_stacked, which takes three required parameters:forward_DAU: the forward DAU projection being visualized (in this case, thefacebook_DAUvariable);forward_DAU_labels: a list of the cohort names as labels for the stacked bars. The length of this list needs to match the number of cohorts in the forward DAU projection;forward_DAU_dates: a list of dates as labels for the X axis. The length of this list needs to match the number of periods in the forward DAU projection;To visualize the Facebook forward DAU projection that reaches the DAU target of 10,000:th.plot_forward_DAU_stacked(forward_DAU=facebook_DAU,forward_DAU_labels=list(facebook_DAU.index),forward_DAU_dates=list(facebook_DAU.columns),)This should produce a graph that looks like this:Note that anything can be provided in theforward_DAU_labelsandforward_DAU_datesparameters. For instance, to give the X axis actual date values (starting from January 1, 2020) and to make the legend more readable, the following can be done:fromdatetimeimportdate,timedeltath.plot_forward_DAU_stacked(forward_DAU=facebook_DAU,forward_DAU_labels=['Cohort '+str(x)forxinlist(facebook_DAU.index)],forward_DAU_dates=[date(2020,1,1)+timedelta(days=int(x)-1)forxinlist(facebook_DAU.columns)])This should produce a graph that looks like this:To create a second retention profile -- this time, for Google -- thecreate_profileprofile can be used again. This time, theprofile_maxparameter will be supplied: whenprofile_maxis provided, the retention profile is projected out to that day (when it is not provided, the retention profile is only projected out to the maximum value provided in thedaysparameter). Also, with the Google retention profile, a much larger dataset of days and retention values will be supplied, so the curve fit is done against many more (arbitrarily produced) data points:importnumpyasnpimportrandomx_data=[1,14,60]y_data=[40,22,10]new_x=[]fori,xinenumerate(x_data):this_x=xforzinnp.arange(1,100):this_y=float(y_data[i]*(1+(random.randint(-20,20)/100)))y_data.append(this_y)new_x.append(this_x)x_data.extend(new_x)google=th.create_profile(days=x_data,retention_values=y_data,profile_max=180)th.plot_retention(google)The output of this should look something like this (the red dots are the actual values fromretention_values):To build a forward DAU projection for Google, the following can be run. Note that the start_date is set to 10:cohorts=[2000,4000,1200,2200,1700,1300,4200,9200]google_DAU=th.project_cohorted_DAU(profile=google,periods=40,cohorts=cohorts,DAU_target=20000,DAU_target_timeline=20,start_date=10)fromdatetimeimportdate,timedeltath.plot_forward_DAU_stacked(forward_DAU=google_DAU,forward_DAU_labels=['Cohort '+str(x)forxinlist(google_DAU.index)],forward_DAU_dates=[date(2020,1,1)+timedelta(days=int(x)-1)forxinlist(google_DAU.columns)])Note the lumpiness of the first few days of DAU -- this is a result of 1) the volatile number of DAU in the initial cohorts and 2) the relatively low Google retention ( 40% on Day 1). Also note the dates on the X axis: the chart starts on January 10th, 2020 since thestart_datevariable is set to 10.In order to get a fuller picture of product DAU, the Facebook and Google forward DAU projections can be combined with thecombine_DAUfunction. The totals for each forward DAU projection should be used, otherwise the graph would be too busy to read:google_total=th.DAU_total(google_DAU)combined_DAU=th.combine_DAU(DAU_totals=[facebook_total,google_total],labels=["Facebook","Google"])th.plot_forward_DAU_stacked(forward_DAU=combined_DAU,forward_DAU_labels=list(combined_DAU.index),forward_DAU_dates=[date(2020,1,1)+timedelta(days=int(x)-1)forxinlist(combined_DAU.columns)])The output of the above should look like:One important aspect of cohort analysis is age segmentation: breaking the user base out into segments based on user age. Theseus comes with two functions to do this:project_aged_DAUandproject_exact_aged_DAU.project_aged_DAUpresents the DAU projection in terms ofminimumuser ages: it can display the number of users that areat leastX days old on a given date.project_exact_aged_DAUpresents the DAU projection in terms ofabsoluteuser ages: it can display the number of users that areexactlyX days old on a given date.Both functions take five parameters:profile: the retention profile being used for the projection;periods: the number of periods for which the forward DAU projection is being made;cohorts: the cohorts that are being projected forward;start_date: the start date of the projection;ages: a list of ages that the projection should be broken down for. Forproject_aged_DAU, the forward DAU projection will be broken out to show the number of users per day that areat leastas old as every age in the list. Forproject_exact_aged_DAU, the forward DAU projection will be broken out to show the number of users per day that areexactlyas old as every age in the list.An example ofproject_aged_DAU:x_data=[1,14,30,90]y_data=[25,18,12,8]#form options: 'log', 'exp', 'linear', 'quad', 'weibull', 'power'snapchat=th.create_profile(days=x_data,retention_values=y_data,profile_max=120)snapchat_aged_DAU=th.project_aged_DAU(snapchat,20,[100,200,300,400,500],start_date=1,ages=[3,7,14])print(snapchat_aged_DAU)This should produce output that looks like the following:12345678910111213141516\age300247114323635234333632732031230529629028270000002265130214320312305296290282140000000000000185510817181920age3275268260254727526826025414180268260254Taking column 10 as an example: 869 users areat least3 days old, 563 users areat least7 days old, and 0 users areat least14 days old.An example ofproject_exact_aged_DAU:snapchat_exact_aged_DAU=th.project_exact_aged_DAU(snapchat,20,[100,200,300,400,500],start_date=1,ages=[3,7,14])print(snapchat_exact_aged_DAU)This should produce output that looks like the following:12345678910111213141516171819\age30024487397121000000000000700000022446688110000000001400000000000001837557493020age3070140Note that each "age" only has five values; this is because there are only five cohorts provided in the example (each cohort will be an exact age only once).Also note that if 1 is passed in theageslist forproject_exact_aged_DAU, it produces a list of DNU:snapchat_exact_aged_DAU=th.project_exact_aged_DAU(snapchat,20,[100,200,300,400,500],start_date=1,ages=[1])print(snapchat_exact_aged_DAU)1234567891011121314151617181920age1100200300400500000000000000000One interesting use case for user base segments is calculating percentages of the overall user base that are at least X days old -- often, certain monetization moments are only available to users after some amount of time, so being able to break a user base out into age groups to know what percentage of the user base is capable of monetizing is helpful. This can be done with Theseus by creating a Total forward DAU projection and combining it with the aged projection:snapchat_DAU=th.project_cohorted_DAU(profile=snapchat,periods=20,cohorts=[100,200,300,400,500],start_date=1)snapchat_total=th.DAU_total(snapchat_DAU)combined_DAU=th.combine_DAU(DAU_totals=[snapchat_aged_DAU,snapchat_total],labels=[["Age "+str(x)forxinlist(snapchat_aged_DAU.index)],"Total"])forxinlist(snapchat_aged_DAU.index):combined_DAU.loc['Age '+str(x)+' Pct']=combined_DAU.apply(lambdaz:(z['Age '+str(x)]/z['Total']))print(combined_DAU)This would output something that looks like:123456\profileAge30.00.024.00000071.000000143.000000236.000000Age70.00.00.0000000.0000000.0000000.000000Age140.00.00.0000000.0000000.0000000.000000Total100.0224.0373.000000545.000000742.000000360.000000Age3Pct0.00.00.0643430.1302750.1927220.655556Age7Pct0.00.00.0000000.0000000.0000000.000000Age14Pct0.00.00.0000000.0000000.0000000.00000078910111213\profileAge3352.0000343.000000336.000000327.000000320.0312.0305.0Age722.000065.000000130.000000214.000000320.0312.0305.0Age140.00000.0000000.0000000.0000000.00.00.0Total352.0000343.000000336.000000327.000000320.0312.0305.0Age3Pct1.00001.0000001.0000001.0000001.01.01.0Age7Pct0.06250.1895040.3869050.6544341.01.01.0Age14Pct0.00000.0000000.0000000.0000000.00.00.0141516171819\profileAge3296.000000290.000000282.000000275.000000268.0260.0Age7296.000000290.000000282.000000275.000000268.0260.0Age1418.00000055.000000108.000000180.000000268.0260.0Total296.000000290.000000282.000000275.000000268.0260.0Age3Pct1.0000001.0000001.0000001.0000001.01.0Age7Pct1.0000001.0000001.0000001.0000001.01.0Age14Pct0.0608110.1896550.3829790.6545451.01.020profileAge3254.0Age7254.0Age14254.0Total254.0Age3Pct1.0Age7Pct1.0Age14Pct1.0In order to actually work with these projections, Theseus comes with two file output functions:to_excelandto_json.to_excelcan take three parameters:df: the forward DAU projection dataframe being output;file_name: the name of the file that will be output (optional)sheet_name: the name of the sheet that the data will be written to (optional)to_excelwill save a .xlsx file in the directory from which the Theseus object is being executed.to_jsoncan take two parameters:df: the forward DAU projection dataframe being output;file_name: the name of the file that will be output (optional)to_jsonwill save a .json file in the directory from which the Theseus object is being executed.ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseMIT