# Double-Constrained Consensus Clustering with Application to Online Anti-Counterfeiting
## 5 Experimenting with Counterfeit Web Shops
### Results  
We tested 64 combinations of clustering methods and features (8 \(\times\) 8) and measured the performance of each using the _F measure_. For ease of interpretation, the results are shown in two distinct charts, rather than as a table. Figure 6 reports the performance of each of the eight clustering methods across the eight clustering features. We also included the performance of a random partition (independent of a feature) for the sake of comparison. The figure clearly suggests that the results of individual methods changed to a great extent as the features varied and that there was no best method across all features, which is consistent with the observation that no clustering method was inherently better or worse than another. More specifically, _Complete_ had the largest performance range (from 0.36 to 0.76), whereas _Centroid_, _Average_, _Single_, _Median_, and _Mcuitity_ were relatively more stable with the change in features (except for the feature _DOM_), with _Ward.D_ and _Ward.02_ exhibiting intermediate behavior. Looking at the relative performance of the clustering methods, _Centroid_, _Average_, _Single_, _Median_, and _Mcuitity_ achieved more comparable and higher results compared to _Complete_, _Ward.02_, and _Ward.D_ (in that order). Finally, every clustering method was markedly better compared to the random partition for each clustering feature. Figure 7 reports the performance of each of the eight clustering features across the eight clustering methods, including the random partition. Analogous to the behavior of clustering methods, the results for individual features changed as the clustering methods varied, and there was no best feature across all methods. Feature _Header120_ exhibited the largest performance range from 0.30 to 0.69, with the other visual features exhibiting similar variations. _DOM_ was also very unstable, whereas the two features pertaining to textual content were relatively more stable across the clustering methods. In terms of feature comparison, _Privacy_ and _Shipping_ usually achieved good results. In contrast, the visual features were, in general, less effective (although _Header90_ performed well for some methods), whereas _DOM_ was comparatively inferior for some methods but also achieved the best overall results for the three methods. Figure 6: Performance of clustering algorithms as a function of clustering features. Figures 6 and 7 show that the performance of all possible combinations of methods and features varied across a very wide range, from a minimum of 0.30 to a maximum of 0.85. As there is no way of knowing in advance which combination will perform better, it is important to find ways to improve the average expected behavior for the set of methods and features at hand. Using consensus clustering and constrained consensus clustering may be an effective strategy, as we now explain. To evaluate _CC_ and _CCC_, the procedure was as follows: For each clustering feature, we used the partitions generated by the eight clustering algorithms as base partitions and computed the _CC_ partition. We then added the constraints and computed the corresponding _CCC_ partitions. We used a growing set of constraints from 10 to 80 (with a step of 10). The full set of 80 constraints used in the experiments was obtained by completing the constraints acquired automatically, as described in Section 5.2.5, with those extracted from CAP using the same method as with the UCI datasets. In all, we generated \(8\times 8\times 8=512\) partitions. Finally, we evaluated the performance of each using the _F measure_. The results are reported in Table 3. For each clustering feature (listed in the first column), we first show the minimum, maximum, and average performance values obtained by the eight _hclust_ algorithms for that feature. In the subsequent columns, we report the performance of CCC using the eight _hclust_ algorithms (with that feature) as input to _CCC_, as the number of constraints grows from 0 to 80.