Package: shapley-decomposition
Shapley DecompositionThis package consists of two applications of shapley values in descriptive analysis: 1) a generalized module for decomposing change over time, using shapley values^1(initially influenced by the World Bank's Job Structure tool^2) and 2) shapley and owen values based decomposition of R^2 (contribution of independent variables to a goodness of fit metric -R^2 in this case-) for linear regression models^3.NotesIdentities or functions with independently moving variables have independent contributions to the result as well. Therefore this module is better useful for functions or identities with dependently moving variables (though it works as well for the independent movements). It should be noted that being able to decompose the contribution of variables doesn't mean that the results are always clearly interpretable. Many features of variables like; scale, dependency mode, change dynamics (slow paced/fast paced, instant/lagged), etc. deserves attention when interpreting their individual contribution to the change or result.Both for the first and second application, the computation time increases exponentially as the number of variables increase. This is the result of powersets and so 2^n  calculations.Shapley value:$v(i) = \sum \limits _{S \subseteq M \setminus i} \phi(s) \cdot [V(S \cup {i})-V(S)]$$\phi(s) = (m-1-s)! \cdot s!/m!$where $i \in M$ and M is the main set of variables and $m=|M|, s=|S|$. For shapley change decomposition, $[V(S \cup {i_{t_1} })-V(S\cup {i_{t_0} })]$ and s is the number of variables with $t_1$ instance.Owen value:$o(i) = \sum \limits _{R \subseteq N \setminus k} \sum \limits _{T \subseteq B_k \setminus i} \phi(r) \cdot \omega(t) \cdot [V(Q \cup T \cup {i})-V(Q \cup T)]$$\phi(r) = (n-1-r)! \cdot r!/n!$$\phi(t) = (b_k-1-t)! \cdot t!/b_k!$where $i \in M$ and M is the main set of variables. N is the powerset of coalition/group set composed of i individuals.  $Q = \bigcup_{r \in R}B_r$ and $n=|N|, r=|R|, b_k=|B_k|, t=|T|$.InstallationRun the following to installpipinstallshapley_decompositionWorkingsshapley_decomposition.shapley_changemodule consists of three functions:samples(),shapley_values()anddecomposition().shapley_change.samples(dataframe)returns powerset pairs that model uses.shapley_change.shapley_values(dataframe, "your function")returns weighted differences for each variable, sum of which gives the shapley value.shapley_change.decomposition(dataframe, "your function")returns decomposed total change by variable contributions. These functions of shapley_change module accepts either or both of thedataandfunctioninputs:The structure of input data isimportant. Module accepts pandas dataframes or 2d arrays:If pandas dataframe is used as input, both the dependent variable and the independent variables should be presented in the given format (variable names as index and years as columns):year1year2yy_valuey_valuex1x1_valuex1_valuex2x2_valuex2_value.........xnxn_valuexn_valueIf an array is preferred, note that module will convert it to a pandas dataframe format and expects y and xs in the following order:[[y_value,y_value],
  [x1_value,x1_value],
  [x2_value,x2_value]]
  ...Identity or function defines the relation between xs and y. Due to the characteristic of shapley decomposition the sum of xs' contributions should be equal to y, i.e. exact (with plus minus 0.0001 freedom in this module due to the residue of arithmetic operations), therefore no place for residuals, or an input relation that fails to create the given y will shoot a specific error. Function input is expected in text format. It is evaluated by a custom syntax parser (as the eval() function has its security risks). Expected format for the function input is the right hand side of the equation:"x1+x2*(x3/x4)**x5""(x1+x2)*x3+x4""x1*x2**2"All arithmetic operators and paranthesis operations are usable:"+" , "-" , "*" , "/" or "Ã·", "**" or "^"Ifshapley_change.decomposition(df,"your function", cagr=True)is called, a yearly_growth (using compound annual growth rate - cagr) column will be added, which will index the decomposition to cagr of the y. Default iscagr=False.Theshapley_decomposition.shapley_r2module consists of three functions as well:samples(),shapley_decomposition()andowen_decomposition.shapley_r2.samples(dataframe)returns powerset variable pairs that model uses.shapley_r2.shapley_decomposition(dataframe)returns the decomposition of model r^2 to the contributions of variables.shapley_r2.owen_decomposition(dataframe, [["x1","x2"],[..]])returns the owen decomposition of model r^2 to the contributions of variables and groups/coalitions. Input features expected by shapley_r2 functions are as:The expected format for the input dataframe or array is:x1x2...xny0x1_valuex2_value...xn_valuey_value1x1_valuex2_value...xn_valuey_value2x1_valuex2_value...xn_valuey_value..................nx1_valuex2_value...xn_valuey_valueshapley_r2.owen_decompositionexpects the group/coalition structure as the second input. This input should be a list of list showing the variables grouped within coalition/group lists. For example a model of 8 variables, x1,x2,...,x8 has three groups/coalitions which are formed as group1:(x1,x2,x3), group2:(x4) and group3:(x5,x6,x7,x8). Then the second input of owen_decomposition should be[["x1","x2","x3"],["x4"],["x5","x6","x7","x8"]]. Even if it is a singleton like group2 which has only x4, variable name should be in a list. If every group is a singleton, then the owen values will be equal to shapley values.As the computation time increases exponentially with the number of variables. For the shapley_decomposition function a default upper variable limit of 10 variables has been set. Same limit applies for owen_decomposition but as the number of groups, not individual variables. However in users' own discretion more variables can be forced by calling the function asshapley_r2.shapley_decomposition(df, force=True)orshapley_r2.owen_decomposition(df, [groups], force=True).ExamplesAs the first influence for the model was from WB's Job Structure, accordingly first example is decomposition of change in value added per capita of Turkey from 2000 to 2018 according to"x1*x2*x3*x4"where x1 is value added per worker, x2 is employment rate, x3 is participation rate, x4 is share of 15-64 population in total population. This is an identity.importpandasfromshapley_decompositionimportshapley_changedf=pandas.DataFrame([[8237.599210,15026.707520],[27017.637990,43770.525560],[0.935050,0.891050],[0.515090,0.57619],[0.633046,0.668674]],index=["val_ad_pc","val_ad_pw","emp_rate","part_rate","working_age"],columns=[2000,2018])print(df)20002018val_ad_pc8237.59921015026.707520val_ad_pw27017.63799043770.525560emp_rate0.9350500.891050part_rate0.5150900.57619part_rate0.6330460.668674shapley_change.decomposition(df,"x1*x2*x3*x4")20002018difshapleycontributionval_ad_pc8237.59921015026.7075206789.1083106789.1083101.000000val_ad_pw27017.63799043770.52556016752.8875705431.3655380.800012emp_rate0.9350500.891050-0.044000-556.985657-0.082041part_rate0.5150900.5761900.0611001285.2000110.189303working_age0.6330460.6686740.035628629.5284100.092726Second example is the decomposition of change in non-parametric skewness of a normally distributed sample, after the sample is altered with additional data. We are trying to understand how the change in mean, median and standard deviation contributed to the change in skewness parameter. Non parametric skewness is calculated by"(x1-x2)/x3", (mean-median)/standard deviation.importnumpyasnpimportpandasfromshapley_decompositionimportshapley_changenp.random.seed(210)data=np.random.normal(loc=0,scale=1,size=100)add=[np.random.uniform(min(data),max(data))forminrange(5,10)]altered_data=np.concatenate([data,add])med1,med2=np.median(data),np.median(altered_data)mean1,mean2=np.mean(data),np.mean(altered_data)std1,std2=np.std(data,ddof=1),np.std(altered_data,ddof=1)sk1=(np.mean(data)-np.median(data))/np.std(data,ddof=1)sk2=(np.mean(altered_data)-np.median(altered_data))/np.std(altered_data,ddof=1)df=pandas.DataFrame([[sk1,sk2],[mean1,mean2],[med1,med2],[std1,std2]],columns=["0","1"],index=["non_par_skew","mean","median","std"])shapley_change.decomposition(df,"(x1-x2)/x3")01difshapleycontributionnon_par_skew0.0658030.044443-0.021359-0.0213591.000000mean-0.247181-0.285440-0.038259-0.0361461.692288median-0.315957-0.333088-0.0171310.016184-0.757719std1.0451881.0720900.026902-0.0013980.065432Third example uses shapley_r2 decomposition with the fish market database from kaggle^4:importnumpyasnpimportpandasfromshapley_decompositionimportshapley_r2df=pandas.read_csv("Fish.csv")#ignoring the species columnshapley_r2.shapley_decomposition(df.iloc[:,1:])shapley_valuescontributionLength10.1948790.220131Length20.1954970.220829Length30.1980970.223766Height0.1168930.132040Width0.1799200.203233#using the same dataframegroups=[["Length1","Length2","Length3"],["Height","Width"]]shapley_r2.owen_decomposition(df.iloc[:,1:],groups)owen_valuescontributiongroup_owenLength10.1575230.177934b1Length20.1581780.178674b1Length30.1602760.181045b1Height0.1410920.159374b2Width0.2682180.302972b2owen_valuescontributionb10.4759770.537653b20.4093090.462347# if the species are included as categorical variablesfromsklearn.preprocessingimportOneHotEncoderenc=OneHotEncoder(handle_unknown="ignore")encti=enc.fit_transform(df[["Species"]]).toarray()df2=pandas.concat([pandas.DataFrame(encti,columns=["Species"+str(n)forninrange(len(encti[0]+1))]),df.iloc[:,1:]],axis=1)shapley_r2.shapley_decomposition(df2,force=True)# as the number of variables bigger than 10, force=Truegroups=[["Species0","Species1","Species2","Species3","Species4","Species5","Species6"],["Length1","Length2","Length3"],["Height","Width"]]shapley_r2.owen_decomposition(df2,groups)#no need for force as the number of groups does not exceed 10