## 2 Data Science Project Organisation--Industry 'Best Practices'
### Data Analysis--Typical Tasks, Terms and Algorithms
\begin{tabular}{c} Supervised learning \\ (both Classification \\ and Regression) In \\  
_Appl. Sci._**2021**, _11_, 12135  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Name of Method** & **Recommended Use** & **Method Type** & **Input Variables** & **Output** & **Reference to Python Implementation** \\ \hline k-means & Clustering is used & & & From sklearn.cluster \\  & when we want to & & & import KMeans \\ DBSCAN & divide our datapoints & & & From sklearn.cluster \\  & into groups according & & & import D8 SCAN \\ Gaussian mixture & k-means algorithm can & & & From sklearn.mixture \\ model & find spherical clusters & & & import \\  & easily, while DBSCAN & & & Gaussian Mixture \\ Mean-shift & can find clusters of any & & & From sklearn.cluster \\  & shape as long as the & & & import MeanShift \\  & dataset has balanced & & & From sklearn.cluster \\  & density. Moreover, & & & import \\  & there are algorithms & & & AffinityPropagation, \\  & which are also good at & & & from sklearn.cluster \\ Others & the clustering of & & & import Agglomera- \\  & datasets with & & &iveClustering, from \\  & unbalanced density, & & & sklearn.cluster import \\  & such as mean-shift. & & & Birch \\ \hline PCA (principal component analysis) & & & & From \\  & & & & sklearn.decomposition \\  & & & & import PCA \\ \hline LDA (linear discriminant analysis) & Dimensionality & & & From \\  & reduction can serve as & & & \\  & data preparation, as the & & & \\  & machine learning & & & \\  & algorithms work better & & & \\  & on low-dimension data. & & & \\  & However, the reduction & & & \\  & can be the main & & & \\  & exercise too. & & & \\ Others & & & & \\  & & & & \\  & & & & \\  & & & & \\ \hline  & One uses Association & & & \\  & when the aim is to & & & \\  & study the connection & & & \\  & between the datapoints, & & & \\  & and make & & (association) & & \\  & recommendation for & & & \\  & new datapoints. & & & \\ \hline  & To compare the means & & & \\  & of 2 groups. It assumes & & & \\  & that the samples are & & & \\  & (approximately) & & & \\ T-test & normally distributed & & & \\  & and are independent, & & & \\  & and have a similar & & & \\  & amount of variance & & & \\  & within each group. & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Cont._  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Name of Method** & **Recommended Use** & **Method Type** & **Input Variables** & **Output** & **Reference to Python** \\ \hline \multirow{8}{*}{F-test} & The hypothesis that the means of a given set of normally distributed populations, all having the same standard deviation, are equal. The hypothesis that a proposed regression model fits the data well. (lack-of-fit sum of squares), & Hypothesis testing & Independent and distributed data with a common variance. & F-value, \(p\)-value, degrees of freedom & From scipystats import f\_oneway \\  & The hypothesis that a data set in a regression analysis follows the simpler of two proposed linear models that are nested within each other. & Independent and normally & & \\ \hline \multirow{8}{*}{Z-test} & One can use Z-test when the sample size is greater than 30, the data points are independent from each other, the data is (approximately) & Independent and normally & & \\  & Other, the data is (approximately) & Hypothesis testing with large (>50) & & \\  & normally distributed, sample sizes are equal. & & & \\  & To check whether & & & \\  & sample mean of the two & & & \\  & groups are equal or not. & & & \\ \hline \multirow{8}{*}{Chi-square Test} & Chi-squared test is used to determine whether there is a statistically significant difference between the expected frequencies and the observed frequencies in one or more categories. & & & \\ \cline{1-1}  & & & & \\ \cline{1-1}  & & & & \\ \cline{1-1} \cline{2-5}  & & & & \\ \cline{1-1} \cline{2-5}  & & & & \\ \cline{1-1} \cline{2-5}  & & & & \\ \cline{1-1} \cline{2-5}  & & & & \\ \hline \hline \end{tabular}
\end{table}
Table 1: _Cont._  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Name of Error Metrics** & **Recommended Use** & **Typical Drawback** & **Input Variable** & **Output Type and Range** & **Reference to Python** \\ \hline Mean absolute error/MAE & & & & & **Implementation** \\ \hline Mean square error/MSE & Regression. & & & & From sklearn.metrics \\ \hline Mean square error/MSE & Gives an absolute measure; can be used to compare & Specific for the given model and dataset. & Continuous & Continuous value, [0; so] & **From sklearn.metrics** \\ \hline Root mean square error/RMSE & regression models on the same dataset. & & & & **From sklearn.metrics** \\ \hline Standard error/residual standard error & & & & & **Import** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Metrics used for evaluation of machine learning models. In this table, we provide a list of useful metrics used when quantifying success in a data analysis experiment, be it a regression or classification problem. As with Table 1, the list is not exhaustive, and interested users are directed to [17; 18; 19] for more information.  
\begin{table}
**Name of Error Metrics** & **Recommended Use** & **Typical Drawback** & **Input Variable** & **Output Type and Range** & **Reference to Python Implementation** \\ \hline R2-value/R squared & Regression. A basic metric to evaluate regression models. & Does not take into account the number of independent variables. & Continuous value, [0; 1] & From sklearn.metrics import r2\_score \\ \hline Adjusted R2/adjusted R square & Regression. Useful to compare models with differing numbers of independent variables. & Continuous value, [0; 1] & Import statsmodels.api \\ \hline Confusion matrix & Classification. Can be used to calculate other classification metrics. & The simple quantities of true positives (TP)/false positives (FN) refer only to the given model and dataset. & Discrete & Matrix for each classes values [0; \(\infty\)] & From sklearn.metrics import confusion\_matrix \\ \hline True positive rate/TPR/recall/ sensitivity/probability of detection & Classification & Not recommended to use alone. & Discrete & Continuous value, [0; 1] & From sklearn.metrics import recall\_score \\ \hline False positive rate/TPR/type I error & Classification & & & & \\ False negative rate/FNR/type II error & Classification & & & & \\ \hline False negative rate/FNR/type II error & Classification & & & & \\ \hline \hline \end{tabular}