# Problem-Driven Scenario Generation for Stochastic Programming Problems: A Survey
## 3 Quality Evaluation of a Scenario-Tree and a Solution
### In-Sample and Out-of-Sample Performance  
Regarding the evaluation of performance in optimization problems, computing time (or tractability in the case of computationally challenging problems) and the quality of a solution serve as typical performance metrics. The evaluation of these performance aspects for a solution derived from a specified scenario-tree using the same scenario-tree is referred to as \"in-sample performance\" evaluation. To derive an accurate objective function value, it is common practice to perform an out-of-sample evaluation, which evaluates the solution's performance on scenarios outside the scenario-tree. This evaluation is crucial in validating models across different fields. As an illustration, the scenario-tree can be conceptualized as a training set in machine learning, wherein the model is trained and learns from the provided data. Correspondingly, the out-of-sample evaluation serves as a validation set, where the model's performance is assessed on unseen data to evaluate its generalization capabilities. In SP problems, a solution is obtained by using a specific scenario-tree. The solution is expected to be feasible and optimal for all cases considered, while keeping a reasonable computing time. Consequently, the size of the scenario-tree is constrained, which can lead to disparities between the in-sample and out-of-sample performance. In consideration of this premise, the work presented in [30] introduces a scenario generation process by minimizing a loss function that measures the discrepancy between out-of-sample and in-sample performance of the solutions. The criterion for a desirable scenario-tree \(\mathcal{T}\) is also established as follows: Given a set of solutions to be evaluated both out-of-sample and in-sample with respect to the actual distribution \(\xi\), in a good scenario-tree \(\mathcal{T}\):  
* The ranking of the solutions is preserved: if solution \(x_{1}\) outperforms \(x_{2}\) when evaluated out-of-sample, it should be highly probable that \(x_{1}\) will also outperform \(x_{2}\) when evaluated in-sample. * Overconfident outliers should be excluded. An outlier is defined as the inconsistency between the out-of-sample \(f(x,\xi)\) and in-sample values \(f(x,\mathcal{T})\). For example, for a solution \(x_{1}\) better than \(x_{2}\) in a maximizing problem:
* If \(f(x_{1},\xi)>f(x_{2},\xi)\), \(f(x_{1},\mathcal{T})<f(x_{2},\mathcal{T})\) while \(f(x_{1},\xi)\approx f(x_{2},\xi)\), it is an acceptable outlier. * If \(f(x_{1},\xi)<f(x_{2},\xi)\), \(f(x_{1},\mathcal{T})>f(x_{2},\mathcal{T})\), it is a non-acceptable outlier. * The in-sample values approximate well the out-of-sample values: \(f(x,\xi)\approx f(x,\mathcal{T})\). The criterion shows that a fundamental metric for assessing the quality of scenario-trees is their capacity to effectively preserve the quality of a solution. The discussion highlights that, when assessing the quality of scenario-trees, the out-of-sample performance carries greater importance than the in-sample performance, particularly when inconsistencies arise between the two. This leads to the stability arguments that address the robustness of the tree. A survey on stability-based approaches for scenario generation can be found in [31]. Several classical methodologies in Stochastic Programming have been discussed, together with a comprehensive review on optimal scenario reduction methods and techniques for generating scenario-trees. Regarding the stability arguments and robustness of scenario-trees, a natural consideration arises that a scenario-tree is good when a substantial number of scenarios are incorporated. This is, however, impractical due to computational efficiency constraints. Therefore, another focus of attention centers on determining the minimum number of scenarios required.