# Automatic Scaffolding Workface Assessment for Activity Analysis through Machine Learning
## 3 Research Design
### Classifier Training  
For the annotation, we collect a series of short video sequences with different durations for each activity category and each video sequence contains one single scaffolder conducting one type of activity. Additionally, based on our definition of scaffolding activities, we labeled these sets of video sequences with their activity categories: scaffold erecting (direct work), transporting (essential contributory work) and idling/waiting (ineffective work). The dataset details are presented in Table 1.  
As shown in Figure 7, we directly obtained 18 3D joint positions from the 3D pose estimation model. These joint points are under absolute 3D coordinates. Since the facial key points are not relevant and essential for the scaffolding activity recognition, for example, the locations of eyes and ears do not provide vital information for activity recognition, we simplify the points of eyes and ears by only keeping the nose location instead. This simplification can effectively increase accuracy and save the computational cost for classification. Since one single point in a 3D space provides x, y and z coordinate information, 14 3D joint points left from the elimination of 4 points of ears and eyes, comprise 42 features. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Activity Category** & **No. of Samples** & **Sum** \\ \hline Class 1: Working & 811 & \\ Class 2: Transporting & 779 & 1731 \\ Class 3: Idling & 141 & \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Dataset details. To facilitate the computation and make all the data comparable, we convert the absolute coordinates into relative coordinates by setting the central point as a zero point (0, 0, 0) and cutting off the same amount of x, y and z values, which are the initial values of the central point, from each key points respectively. We adopt the pose codebook that each pose of a kind of activity is viewed as a single histogram containing 42 features or vectors. To recognizing scaffolding activity from a short video sequence, a discriminative classifier is trained with the annotated data. The classifier takes the body skeleton features as the input, which provides sufficient descriptive content for activity classification. At the training stage, firstly we extract key points of a skeleton from every video sequence for each activity category and each video sequence is processed into frames which become the input of our pose estimation model. For every image frame, a set of 42 features generated by the pose estimator, plus one manually labeled feature indicating the activity category, constitute the input for supervised classifiers. Since different supervised classifiers have various performances on one particular classification task, we introduce major popular multi-class algorithms including Random Forests (RF), Decision Tree (DT), Artificial Neural Networks (ANN), One-vs.-One Support Vector Machine (One-vs.-One SVM), One-vs.-All Support Vector Machine and K Nearest Neighbors (KNN) [89]. These algorithms are capable of multi-class classification as well as handling data with multiple features [89; 90; 91]. To select a classifier with outstanding performance, each aforementioned algorithm is employed for classification through our dataset of scaffolding activities and the training and testing process follows the principle of cross-validation which is a statistical procedure for the evaluation and validation of machine learning models. **Decision Tree (DT). ** Just like its name, DT is a kind of algorithm with a tree structure that a tree leaf denotes an outcome label, and a branch represents a sub-section of an entire tree or a sub-tree. DT can be used for classification and regression. For classification, a tree is constructed through the process of binary recursive partitioning, which is a procedure iteratively splitting the data into partitions.