## 3 Public Datasets for Scene Understanding Methods
### Solutions to Handle Dataset Imbalance
Therefore, splitting the data before applying oversampling techniques is mandatory to ensure accurate evaluation and generalization of the model. **Combined overand undersampling** can be a solution to the overfitting problem. [PERSON]. [PERSON] et al. [98] combined SMOTE oversampling with Tomek links undersampling as a cleaning technique by removing overlapping samples in the training set to improve classification in a binary case. However, active sampling may not be the most effective approach to deal with imbalanced 3D point cloud data. Sampling techniques have proven to be effective for binary datasets [97] and datasets with a small number of features [99], but point clouds typically hold multiple features (e.g., X, Y, and Z-coordinates and r, g, and b colors for each point), which can make it difficult to pick meaningful samples or to determine which samples to discard. Passive methods, which keep the dataset unchanged, include penalizing misclassification through **cost-sensitive learning**, adjusting the **decision threshold** of the classifier, and changing the **performance metric** to help the learner deal with the imbalance without the need to change the training dataset configuration. Accuracy and error rates are easy to implement metrics to evaluate the performance of multi-class classifiers and are frequently used in point cloud classification [17,100,101]. However, using singular assessment criteria does not provide adequate information, particularly in the context of imbalanced datasets [102], where the classifier prefers to ignore minority classes in favor of high accuracy on the majority classes. An exemplary case can be found in the situation when the majority class represents 99% of all cases, and the classifier simply assigns the label of the majority class to all test cases. An excellent but misleading accuracy of 99% will be assigned to a classifier that classified no minor cases right [103]. More informative assessment metrics are necessary for conclusive evaluations of performance in the presence of imbalanced data. In general, the machine learning literature often proposes the following evaluation metrics: receiver operating characteristics (ROC) curves, the area under the ROC curve (AUC), precision-recall (PR) curves, and cost curves, to evaluate the performance of imbalanced data classification from the confusion matrix [91]. However, these graphical methods are designed for binary classification and cannot be applied to multi-class classification problems. In the case of multi-class classification, the confusion matrix must be binarized. This can be conducted in two different ways: the _One-vs-Rest scheme_ compares each class against all the others (assumed as one) and the _One-vs-One scheme_ compares every unique pairwise combination of all classes [104]. State-of-the-art benchmarks use multi-class-suited metrics to compare performance in 3D point cloud tasks in practice. The most relevant ones are listed in Table 3 and application-orientedperformance metrics will be discussed in Section 4. Finally, a confusion matrix (binary case in Figure 9 and multi-class in Figure 7c) is a powerful tool to visualize classification results. **Cost-sensitive learning** considers the cost of prediction errors and assigns penalties to each class through a cost matrix. Increasing the cost of the minority group is equivalent to increasing its importance and decreases the likelihood that the learner will misclassify instances from this group [90,111]. A compilation of cost-sensitive learning approaches for neural networks is given by [PERSON] et al. [103]. While implementing cost-sensitive learning into the training algorithm is a comparatively easy task and most common algorithms are modified to take a class penalty or weight into consideration, one of the biggest challenges in cost-sensitive learning is the assignment of an effective cost matrix [90]. The cost matrix can be defined empirically based on past experiences or by a domain expert with knowledge of the problem.