## 2 Materials and Methods
### Overview of the TID2013 Dataset and Earlier Results in the Combined Metric Design
Another subset (Actual) contains images with aforementioned distortions ##1, 3, 4, 5, 6, 8, 9, 19, and 21, as well as #10 (JPEG compression) and #11 (JPEG2000 compression). Some examples of when these types of distortion might happen in remote sensing are discussed in our earlier paper [11]. Thus, being united, the two subsets can represent a variety of distortions typical for remote sensing. It is worth noting here that some other types of distortions may also be present in RS images. For example, there may be multiple distortions such as blur + noise or blur + blockiness. However, currently, the analysis is restricted by the availability of the databases and appropriate subsets. Previous experiments with the design of the combined FR and NR metrics [11; 24; 29; 30] show the following:  
* SROCC for a given metric, elementary or combined, depends not only on a metric but also on the database whereas for databases with a smaller number and/or more typical distortions SROCC is usually considerably higher (because of this, the metric performance for the old LIVE database is commonly much better than for the other, more complicated, databases);
* SROCC values for metrics for such subsets of TID2013 as Noise or Actual are usually larger than for the entire database;
* Combined metrics, especially NN-based, can provide better results than the best elementary metric; meanwhile, for a given number of elementary metrics \(N\), it does not mean that the best \(N\) elementary metrics have to be combined to produce the best NN-based metrics; the elementary metrics being complementary to each other constitute the best solutions. Here it is worth recalling that there are three main types of combining elementary metrics:
* CM = R(EMfMOS(\(n\)), \(n=1\),..., \(N\)), where R means a robust operator (e.g., a sample median or \(\alpha\)-trimmed mean) applied to a set of elementary metrics after fitting to MOS (EMfMOS), \(N\) is the number of elementary metrics which is usually quite small (e.g., equal to 5); the advantage is that the method is simple; the drawback is that metric to MOS fitting is needed;
* CM = WSUM(EM(\(n\)), \(n=1\),..., \(N\)) or CM = WPROD(EM(\(n\)), \(n=1\),..., \(N\)) where WSUM and WPROD denote the weighted sum or weighted product of \(N\) elementary metrics with weights optimized to provide the highest correlation with subjective scores for a given dataset; the advantages are that good results may be obtained for a relatively small number of elementary metrics and their combining is very simple and fitting (in general) is not needed; the drawback is that the obtained combined metrics are usually less efficient than the NN-based ones;
* CM = NN(EM(\(n\)), \(n=1\),..., \(N\)) where NN(\(0\)) means that a neural network is applied to a set of input parameters where elementary metrics (without fitting) serve as inputs; the advantage is that the obtained metrics are usually the most efficient; the drawback is that there are several questions to be answered at the stage of metric design. Because of the mentioned advantages and drawbacks, this paper concentrates on the two latter approaches to the design of combined metrics. In both cases, the provided performance depends upon many factors. The main two of them are the number of elementary metrics and the choice of the used elementary metrics. A general tendency is that a larger number of elementary metrics leads to better performance but only to a certain limit. For the combined metrics presented as a weighted sum or product [29; 30], it is often enough to have less than 8-10 metrics since a further increase of \(N\) does not lead to sufficient benefits in performance, making the combined metric more complicated. The same holds for NN-based metrics [11] where usually it is enough to apply 20-30 elementary metrics instead of 40-50 elementary metrics used as inputs. In this case, the Lasso method [34] allows restricting a set of elementary metrics to be employed [11] with simplifying the design and the final structure of the obtained NN-based combined metric.