Voronoi Tessellation-based Confidence Decision Boundary Visualization to Enhance Understanding of Active Learning

Authors: Jie Chen, Siteng Ma, Dairui Liu, Brian Mac Namee, Ruihai Dong
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: Decision Boundary, Visualization, Active Learning

Abstract:
The current visualizations used in active learning fail to capture the cumulative effect of the model in the active learning process, making it difficult for researchers to effectively observe and analyze the practical performance of different query strategies. 
To address this issue, we introduce the confidence decision boundary visualization, which is generated through Voronoi tessellation and evaluated using ridge confidence. This allows better understanding of selection strategies used in active learning. This approach enhances the information content in boundary regions where data distribution is sparse. Based on the confidence decision boundary, we created a series of visualizations to evaluate active learning query strategies. These visualizations capture nuanced variations regarding how different selection strategies perform sampling, the characteristics of points selected by various methods, and the impact of newly sampled points on the model. This enables a much deeper understanding of the underlying mechanisms of existing query strategies.

Decision: reject
Reviewer scores: 6, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730554820580

thread_type:
official_review

<thread_object>
creator: reviewer_rwsq

<content>
summary:
This paper presents a Voronoi tessellation-based method to visualize decision boundaries in active learning. By highlighting ridges between cells with differing predictions, this method provides a clearer view of the decision boundaries. In addition, it introduces a ridge confidence metric to quantify prediction uncertainty. Experiments on MNIST and CIFAR-10 illustrate the effectiveness of this approach for analyzing various active learning strategies, providing insights into the behavior and effectiveness of different sampling techniques.

strengths:
1. Comprehensive Comparison of Active Learning Strategies: The evaluation includes a systematic comparison of various active learning strategies. These potentially contribute to the broader discourse on active learning methods and offer practical guidance for future research in this area.
2. Effective Use of Visualization Techniques: The paper incorporates visualization methods to facilitate the analysis of data and models. This makes the analysis and findings more accessible and intuitive.

weaknesses:
1. **Concerns about Decision Boundary Generation**. From the figures, the projection results and the Voronoi tesselation results appear to be fixed across multiple rounds. However, as feature vectors update during training, the projection results and the decision boundaries should also be updated. It is essential to clarify how well these fixed tessellations with the updated predictions capture the model's behavior. In addition, the use of 2D projections to represent high-dimensional decision boundaries raises concerns, as results can vary significantly based on the selected projection method and parameters.
2. **Insufficient evaluation**. While this paper compares different active learning strategies and summarizes some insights, this evaluation is not sufficient and rigorous. On the one hand, the evaluation is only conducted on MNIST and CIFAR-10, which only contain a few classes with substantial differences between them. It remains uncertain how the proposed method performs with more classes or finer distinctions. On the other hand, the evaluation of boundary effectiveness is inadequate. Many insights, such as oversampling on the boundary region in the uncertainty-based methods, can be identified from scatterplots alone, making the Voronoi approach seem unnecessary.
3. **Omission of Relevant Literature**. The paper overlooks significant works that utilize Voronoi tessellation for visualizing sample boundaries. For instance, Chen et al. [*] use it to visualize the samples and boundaries in the analysis of label propagation. Including such references would enhance the contextual foundation of the research.
[*] Interactive Graph Construction for Graph-Based Semi-Supervised Learning. TVCG 2021.

questions:
1. Decision Boundary Representation: How do the fixed Voronoi tessellation results relate to the updated feature vectors during training? Can you explain how this approach ensures an accurate representation of the model's decision boundaries throughout the training process?
2. Generalizability: How does the method work on more complicated datasets, such as ImageNet (with more classes) and CUB-200-2011 (fine-grained classification tasks)
2. Boundary Effectiveness: Can you elaborate on how the Voronoi tessellation approach adds value beyond insights obtainable from scatterplots? What specific contributions does it provide in terms of understanding boundary effectiveness that traditional methods do not?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730564020976

thread_type:
official_review

<thread_object>
creator: reviewer_8shq

<content>
summary:
This paper introduces the confidence decision boundary visualization method, which provides an interpretable comparison of various active learning strategies, yielding valuable insights. The experimental design and discussion are thorough.
I’m glad to see that some work at the ICLR conference focuses on data visualization. The insights provided by data visualization can better help users understand some of the underlying aspects behind the models. Additionally, visualization on the existing dataset is able to provide better insights and differentiation capabilities.

One aspect I am particularly concerned about is the scalability of this visualization. Currently, the data volume is relatively small, but with a large number of categories, how can the scalability of this visualization be ensured? I would suggest that the authors provide more discussion and experimental results on this point. When the dataset is large, the entire Voronoi becomes complex, with very small cells and densely packed ridges that obscure each other, making it visually unfriendly. I doubt whether users other than authors can derive similar insights from such dense and non-interactive visualizations. The authors could provide quantitative and qualitative user surveys to demonstrate the effectiveness and usability of their method. The authors could consider further optimizations, such as clustering before segmentation or refining the segmentation rules, rather than just using Voronoi tessellation.
The visualization method seems to lack interactivity. Providing an interactive tool would enhance the appeal of this work. Additionally, the authors do not explicitly mention which dimensionality reduction method was used. Different dimensionality reduction techniques may affect subsequent Voronoi and decision boundary construction. The authors should provide explanations and evaluations of this aspect.

strengths:
I’m glad to see that some work at the ICLR conference focuses on data visualization. The insights provided by data visualization can better help users understand some of the underlying aspects behind the models. Additionally, visualization on the existing dataset is able to provide better insights and differentiation capabilities.

weaknesses:
One aspect I am particularly concerned about is the scalability of this visualization. Currently, the data volume is relatively small, but with a large number of categories, how can the scalability of this visualization be ensured? I would suggest that the authors provide more discussion and experimental results on this point.

questions:
One aspect I am particularly concerned about is the scalability of this visualization. Currently, the data volume is relatively small, but with a large number of categories, how can the scalability of this visualization be ensured? I would suggest that the authors provide more discussion and experimental results on this point. When the dataset is large, the entire Voronoi becomes complex, with very small cells and densely packed ridges that obscure each other, making it visually unfriendly. I doubt whether users other than authors can derive similar insights from such dense and non-interactive visualizations. The authors could provide quantitative and qualitative user surveys to demonstrate the effectiveness and usability of their method. The authors could consider further optimizations, such as clustering before segmentation or refining the segmentation rules, rather than just using Voronoi tessellation.
The visualization method seems to lack interactivity. Providing an interactive tool would enhance the appeal of this work. Additionally, the authors do not explicitly mention which dimensionality reduction method was used. Different dimensionality reduction techniques may affect subsequent Voronoi and decision boundary construction. The authors should provide explanations and evaluations of this aspect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569442696

thread_type:
official_review

<thread_object>
creator: reviewer_vpbn

<content>
summary:
The proposed work deals with the challenging issue of getting insights into sampling strategies of active learning. Addressing this issue, the authors developed a novel visualization approach: The feature space of the samples is projected to a 2D plane and segmented into Voronoi cells. Ridges of the Voronoi cells are used to construct the decision boundary. Additionally, the confidence of this decision boundary is assessed by leveraging predicted probabilities of samples for each ridge. Thus, the confidence of the decision boundary varies locally. The authors apply the confidence decision boundary to visualize the learning and sampling behavior of eight active learning strategies on MNIST and CIFAR-10. Using various other visualizations, they qualitatively depict the characteristics of the sampling strategies and conclude that uncertainty caused by insufficient training samples may be effectively tackled by sampling. In contrast, uncertainty in noisy regions may be hard to tackle.

strengths:
- The paper sheds light on the crucial topic of getting insights into machine learning algorithms and provides visual results that highlight apparent differences between sampling strategies.
- The chosen approach of this paper is generalizable to various machine learning models performing prediction tasks.
- The authors provide extensive visualizations for each sampling strategy and compare their characteristics with the help of side-by-side visualizations.
- The paper examines a high number of sampling strategies.

weaknesses:
- The paper contains errors in writing. For example, the sentence "Since points on either side of the predicted ridges belong to different predicted classes, the confidence of the predictions vary, different sections of the decision boundary carry varying degrees of informative value due to differences in prediction confidence." in lines 110-113 should be corrected.  
- Visualizations lack proper color encoding. For example, the decision boundary, a continuous value, is represented with a qualitative color scale (see Figure 5).
- It remains unclear which snapshots of the feature space were taken for visual elaboration. For example, it is not described why the third round of training was used to inspect the decision boundary in Figure 5. Thus, the authors' conclusions are only based on single visual findings, making them hard to retrace.
- A clear methodology for selecting snapshots for inspection would improve the clarity and usefulness of the authors' approach. Furthermore, the evaluation relies on observations of a single training run - a more extensive evaluation with a higher number of training runs per sampling strategy would be a good starting point.
- Some visualizations lack proper annotations, making them hard to understand. Figure 4 especially needs annotations on what the different colors encode. Similarly, which training rounds do we see in Figure 2?

questions:
- It is unclear why one can assess the confidence of the predicted ridges with the representative Voronoi center points $\mathbf{p_i}$ and $\mathbf{p_j}$ (equation 2). Although each ridge can be represented by its representative point $\mathbf{p}$, the predicted probabilities close to the decision boundary may differ strongly from their center. Could you please clarify why using the point $\mathbf{p}$ gives an accurate prediction of the confidence?
- How did you decide on which snapshots to visualize?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730572109413

thread_type:
official_review

<thread_object>
creator: reviewer_kbv3

<content>
summary:
This paper introduces a Voronoi tessellation-based visualization method for active learning (AL) to address the limitations of traditional visualizations, which often lack detail in uncertain or sparse boundary areas. By using a ridge confidence metric with Voronoi cells, the approach offers a clearer, more informative view of decision boundaries, aiding in analyzing different AL sampling strategies. Tested on MNIST and CIFAR-10 datasets, the method reveals unique sampling behaviors across entropy, margin, and diversity-based strategies, helping clarify how each handles uncertainty and impacts model learning.

strengths:
1. By using Voronoi tessellation and introducing a ridge confidence metric, the paper provides a more detailed and interpretable visualization of decision boundaries, offering insights into the complexity and uncertainty of boundary regions.
1. The authors compare multiple AL sampling methods (e.g., entropy, margin, BALD dropout, KMeans) on distinct datasets, providing valuable insights into the behaviors and trade-offs of each approach in different scenarios.
1. The visualization effectively demonstrates how models handle uncertainty due to limited training samples and noisy regions, which is beneficial for identifying optimal query strategies for different types of uncertainties.
1. While focused on active learning, the proposed visualization technique has potential utility in other areas of machine learning that require understanding complex decision boundaries in high-dimensional spaces.

weaknesses:
1. This paper lacks a review of related work. It's important to examine the literature in the related fields, especially other visualization methods developed for active learning.
1. While the work in the paper is valuable, it also lacks of the baseline methods. The paper demonstrates the effectiveness of Voronoi tessellation and ridge confidence in active learning throught several case studies, but it does not prove how much better it is compared with other visualization methods. I suggest a quantitative analysis with different visualization methods, like a user study.
1. Section 3 is over lengthy. The authors should consider breaking it down. Subsection 3.1 and Subsection 3.2 should be independent sections.
1. The discussion of the 2D feature map is missing. How much does space distortion affect the Voronoi tesselletion construction? The neighbors in the original space are not always the same in the feature space. How reliable is the decision boundary in this case? Since everything is visualized in the feature map, it's essential to give a comprehensive discussion on the feature map itself.
1. The visualization strategies are informative, but the implementation in the plots are really hard to follow. I have several suggestions for improvement:
    - Figure 2, 3, 5, 6, 7, and 8 are visualization results, but they are too small and dense to read, which is fatal for a visualization paper. At least the authors should put high resolution images in the appendix.
    - In figure 5, the colormap of confidence interval should be different from that of the classes. I suggest different levels of gray to show the confidence interval
    - In the visualization plots, the scatter points, Voronoi ridges, and cells are crowded and overlapping, causing severe visual clutter. Actually, it's not necessary to show all the information in a single plot at once. It only makes the plot massive. One way to display a large quantity of information is to enable user interactions. For example, enable users to choose classes they are interested in, add a slider to show different levels of confidence, brush to zoom in on a decision boundary. 
    - The Error Detection visualization results are interesting. It gives more information on how the active learning model behaves. I suggest putting multiple training iterations of Error Detection plots in the appendix to visualize its evolution over time. 

While I believe this work makes good contribution, I'm afraid the authors cannot resolve my concerns in a reasonable time. Therefore, I recommend a weak reject.

questions:
I don't quite understand how the 2D feature map is constructed. It is said that it directly comes from the neural network model, but how? Could the authors provide more details?
</content>
</thread_object>

</thread>


Metareview:
metareview:
**(a) Summary**

This paper presents a novel visualization approach for active learning using Voronoi tessellation and a ridge confidence metric. The method projects high-dimensional feature spaces onto 2D planes and constructs decision boundaries using Voronoi cell ridges. The authors apply this visualization technique to analyze eight different active learning sampling strategies on MNIST and CIFAR-10 datasets, revealing distinct sampling behaviors and their effectiveness in handling different types of uncertainties.

**(b) Strengths**
- Provides detailed and interpretable visualization of decision boundaries through Voronoi tessellation
Effectively demonstrates model behavior in handling different types of uncertainties
- Successfully provides insights into ML models through comprehensive comparison of sampling strategies

**(c) Weaknesses**
- Limited literature review and baseline comparisons
- Lack of user studies to validate visualization effectiveness
- Scalability challenges with larger datasets
- Suboptimal visualization design choices, particularly in color encoding

**(d) Decision factors** 

While the paper presents an interesting visualization approach for active learning, several key issues remain unresolved. The visualization design choices need refinement, particularly in color encoding and scalability. The lack of user studies and baseline comparisons makes it difficult to validate the method's practical utility. While we acknowledge the potential value of this approach, these limitations lead to a recommendation for rejection.