SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chunlin Yu<sup>1</sup> Ye Shi<sup>1,2</sup> Jingya Wang<sup>1,2*</sup>
$^{1}$  ShanghaiTech University
$^{2}$  Shanghai Engineering Research Center of Intelligent Vision and Imaging
{yuchl,shiye,wangjingya}@shanghaiitech.edu.cn
Previous endeavors in self-supervised learning have enlightened the research of deep clustering from an instance discrimination perspective. Built upon this foundation, recent studies further highlight the importance of grouping semantically similar instances. One effective method to achieve this is by promoting the semantic structure preserved by neighborhood consistency. However, the samples in the local neighborhood may be limited due to their close proximity to each other, which may not provide substantial and diverse supervision signals. Inspired by the versatile re-ranking methods in the context of image retrieval, we propose to employ an efficient online re-ranking process to mine more informative neighbors in a Contextually Affinitive (ConAff) Neighborhood, and then encourage the cross-view neighborhood consistency. To further mitigate the intrinsic neighborhood noises near cluster boundaries, we propose a progressively relaxed boundary filtering strategy to circumvent the issues brought by noisy neighbors. Our method can be easily integrated into the generic self-supervised frameworks and outperforms the state-of-the-art methods on several popular benchmarks. Code is available at: https://github.com/cly234/DeepClustering-ConNR.
Fueled by the expressive power of neural networks, deep clustering has emerged as a prominent solution in the low-label regime, where traditional clustering can be significantly enhanced by utilizing latent data representations. Recent advancements in self-supervised learning have further expanded the possibilities for deep clustering [38, 17, 41], providing a solid foundation for grouping similar instances. Among the various self-supervised recipes, contrastive learning methods ensure instance discrimination backed up with sufficient negative samples, while non-contrastive learning methods circumvent the class collision issue [37, 17] by simply pulling two augmented views closer. However, capturing the intrinsic semantic structure with clear decision boundaries remains a challenge in deep clustering, which has led researchers to focus on more group-aware and discriminative approaches.
Group-aware methods commonly enforce the prediction consistency between samples and their nearest neighborhoods, measured by cosine similarity or Euclidean distance for deep clustering[42, 21, 55, 56]. Although these methods are simple and effective, a significant portion of the retrieved neighborhoods may contain semantically redundant information. This is primarily due to the fact that the local neighboring samples are already close to the anchor sample in the feature space, and as such, may not provide sufficient supervision signals for deep clustering. To address this limitation, previous works have added memory banks with historical features to enlarge the search space [21, 49], or labeled the proximity between samples over the entire dataset [55, 56]. Building on the foundation of
Figure 1: (a) Comparison between the conventional Euclidean Neighborhood and the ConAff Neighborhood, both with cross-view consistency. The Euclidean distance retrieves the Euclidean neighborhood, while the contextual distance is used for the ConAff neighborhood. In reciprocal relations, "1" means two samples are in each other's top- $k$  neighbors, and "0" indicates neither is in the other's top- $k$ . The goal is to use reciprocal relations as a contextual distance metric for the ConAff neighborhood. For instance, distant pairs A and B might be contextually similar due to their similar reciprocal relations. (b) Images in the first column and their 10 nearest neighbors in the other columns, were retrieved using Euclidean Neighborhood and the proposed ConAff Neighborhood. Wrong neighbors are marked in red, and hard positives are marked in green. By default, unmarked neighbors are regarded as true neighbors.
previous group-aware methods, a key research question arises: Can we excavate more informative and reliable neighbors from intra-batch samples?
Inspired by the versatile re-ranking techniques that have shown remarkable performance in image retrieval tasks [57, 45, 32, 44, 45], we propose a novel approach to deep clustering that involves retrieving neighborhoods in a contextually affinitive metric space. Typically, the neighborhood information is retained in the ranking list of pairwise similarities between a query sample and other samples. However, such a list may neglect the contextual statistics that exist between the samples. When two images are contextually affinitive, they may share similar relationships to a set of reference images. In this study, the reference images are defined as the current batch, and the relationship between two images is established based on whether they are among each other's top- $k$  neighbors. Consequently, distant image pairs in the original feature space may be contextually affinitive, and therefore considered as more informative positive pairs. Additionally, the contextually affinitive neighborhood in one view may align with the query sample in the other view, thus enabling cross-view neighborhood consistency. Figure 1 (a) presents a comparison between the conventional Euclidean Neighborhood and the proposed ConAff Neighborhood. Additionally, Figure 1 (b) illustrates the results obtained when retrieving the nearest neighbors using both types of neighborhoods.
In the context of learning with neighborhoods, intrinsic noises can exist that are less observable compared to external noises caused by label contamination. These noises originate from the inferiority of the intrinsic feature space and taking these noisy samples as candidates for clustering can lead to the accumulation errors and degrade overall performance. To address this challenge, we propose a progressive boundary filtering strategy based on the ConAff neighborhood that further improves the robustness of clustering. Our strategy filters out boundary samples in a self-paced learning paradigm and gradually involves more complex samples for clustering.
- We propose a Contextually affinitive Neighborhood Refinery framework (CoNR) for deep clustering, which attempts to excavate more semantically relevant neighbors in a contextually affinitive space, termed as ConAff neighborhood.
- We devise a progressive boundary filtering strategy to combat the intrinsic noises existing in the ConAff neighborhood, further improving the robustness of clustering.
- Extensive experiments demonstrate that CoNR improves upon the SSL baseline by a significant margin, proving to be competitive with state-of-the-art methods on five widely-used benchmarks. CoNR's simplicity and effectiveness allow it to be easily integrated into other SSL frameworks, resulting in consistent performance gains.
Our goal is to group a dataset containing  $n$  unlabeled samples into  $k$  semantic clusters, where  $k$  refers to the cardinality of class label sets. Since it is non-trivial to directly obtain clean cluster assignments from scratch, we adopt an unsupervised pre-text task as the initialization for image clustering, which provides a decent neighborhood structure. However, directly utilizing the neighboring information could lead to sub-optimal results as i) the nearest-neighbor searching schema based on cosine similarity may not favor hard positives that are important for clustering, and ii) the top-ranked neighbors of samples located at a cluster boundary contain inevitable noise when the learned clusters have not been not mutually separated. We, therefore, propose our main algorithm for clustering in a contextually affinitive neighborhood, with progressive relaxation of boundary sample filtering, in an attempt to alleviate the above issues.
In general, given a dataset  $\mathcal{D} = \{\pmb{x}_1,\pmb{x}_2,\dots ,\pmb{x}_n\}$ , an image  $\pmb{x}_i$  uniformly sampled from  $\mathcal{D}$ , and two augmentations  $t\sim \mathcal{T},t'\sim \mathcal{T}'$ , we aim to first encourage the instance-aware concordance based on BYOL [11] for early pre-training, and then encourage the group-aware concordance for later finetuning, in which the cornerstone of our method lies. Following [11], the model is equipped with an online network  $q(\mathcal{F}_o(\cdot))$  and a target network  $\mathcal{F}_t(\cdot)$ , where each of  $\mathcal{F}_o(\cdot),\mathcal{F}_t(\cdot)$  consists of an encoder cascaded by a projector, and  $q(\cdot)$  constitutes an additional non-linear predictor.
2.1.1 Instance-aware Concordance
The commonly-used instance discrimination approach for non-contrastive self-supervised learning enforces the similarity between a referenced feature and its corresponding feature in another augmented view, which can be formulated as:
$$
\mathcal {L} _ {s i m} ^ {I} = - \underset {\boldsymbol {x} _ {i} \in \mathcal {B}} {\mathbb {E}} \left[ \right. \langle q \left( \right.\mathcal {F} _ {o} (t (\boldsymbol {x} _ {i})), \mathcal {F} _ {t} \left(t ^ {\prime} (\boldsymbol {x} _ {i})\right) \rangle \left. \right], \tag {1}
$$
where  $\langle \cdot \rangle$  denotes the cosine similarity. This can be interpreted as the promotion of instance-aware concordance under different augmentations and hidden transformations, which could help bootstrap an over-clustered representation via the stop-gradient and momentum-update techniques. Notably, this instance-aware concordance produces a general-purpose representation that can be well transferred to various downstream tasks, as well as a solid basis for neighborhood discovery and higher-order contextual exploration.
2.1.2 Group-aware Concordance
In order to develop the discrimination framework from instance level to group-aware concordance, we consider regarding the local neighborhood as a semantic group that involves more diverse positive samples, sharing the same spirit with previous works [10, 21, 42, 29]. Typically, group-aware concordance can be formulated as:
$$
\mathcal {L} _ {s i m} ^ {G} = - \underset {\boldsymbol {x} _ {i} \in \mathcal {B}} {\mathbb {E}} [ \underset {\boldsymbol {x} _ {j} \in \mathcal {N} (\boldsymbol {x} _ {i})} {\mathbb {E}} [ \langle q (\mathcal {F} _ {o} (t (\boldsymbol {x} _ {i})), \mathcal {F} _ {t} \left(t ^ {\prime} (\boldsymbol {x} _ {j}))\right) \rangle ] ], \tag {2}
$$
where  $\mathcal{N}(\pmb{x}_i)$  denotes the set of neighborhood samples of  $\pmb{x}_i$  preserved in a Euclidean metric space, and  $\pmb{x}_i\in \mathcal{B}$  reveals that all samples in the batch are uniformly considered to encourage the cross-view
neighborhood consistency. While the de-facto methods [10, 21, 42, 29] exploit neighborhoods to enrich the latent representation, they somehow ignore the quality of neighborhoods themselves. On the one hand, good neighbors should be clean, which means that we should select proper candidates from  $\mathcal{B}$  to retrieve clean neighbors. On the other hand, good neighbors should be rich, which motivates us to replace  $\mathcal{N}(\boldsymbol{x}_i)$  with more informative neighborhoods in higher-order space. Therefore, we seek to further improve the group-aware concordance paradigm from two perspectives, which will be introduced in Sec. 2.2 and Sec. 2.3.
2.2 Contextually Affinitive Neighborhood Discovery
As discussed in Sec. 2.1, previous works retrieve the nearest neighbors based on pairwise cosine similarity, where the distance metric depends solely on two involved data points on the hypersphere. On the contrary, our solution is to redefine how two data points are similar by considering contextual relations among the whole batch data. Concretely, we first transform the data points into a contextually affinitive space, where we assume that the more similar two transformed data points are, the more high-order contextual information they share, and thus we can utilize this information to excavate more meaningful neighbor pairs, which we term as the contextually affinitive neighborhood discovery.
Formally, given a batch of normalized features  $\mathcal{B}_f = [f_0,\dots ,f_{|\mathcal{B}_f| - 1}]^T$ , our goal is to first transform  $\mathcal{B}_f$  into a contextually refined feature space  $\mathcal{H}_f = [h_0^r,\dots ,h_{|\mathcal{B}_f| - 1}^r ]^T$ , and then retrieve the nearest neighbors set using the distance metric defined by  $\mathcal{H}_f\mathcal{H}_f^T$ . Inspired by [53, 57, 34] which build the contextual relations over the entire dataset, we efficiently retrieve the contextually affinitive neighborhood via topology graphs in an online manner following [53].
Building Contextual  $k$ -NN Graph. The underlying principle of building the contextual  $k$ -NN graph is to inject contextual information into nodes  $\mathcal{V}$  and edges  $\mathcal{E}$  so that the message propagation can move beyond the local region [53]. First, an online reciprocal adjacency matrix  $A \in \mathbb{R}^{|\mathcal{B}_f| \times |\mathcal{B}_f|}$  is computed as follows:
$$
\boldsymbol {A} _ {i j} = \left\{ \begin{array}{l l} 1, & \text {i f} \boldsymbol {f} _ {i} \in \mathcal {N} (\boldsymbol {f} _ {j}, k _ {1}) \wedge \boldsymbol {f} _ {j} \in \mathcal {N} (\boldsymbol {f} _ {i}, k _ {1}) \\ 0, & \text {i f} \boldsymbol {f} _ {i} \notin \mathcal {N} (\boldsymbol {f} _ {j}, k _ {1}) \wedge \boldsymbol {f} _ {j} \notin \mathcal {N} (\boldsymbol {f} _ {i}, k _ {1}) \\ 0. 5, & \text {o t h e r w i s e} \end{array} , \right. \tag {3}
$$
where  $\pmb{f}_j\in \mathcal{N}(\pmb {f}_i,k_1)$  denotes  $\pmb{f}_{j}$  is among the  $k_{1}$  nearest neighbors of  $\pmb {f}_i$  in  $\mathcal{B}_f$ , and each row of  $\pmb{A}$  can be interpreted as the  $k$ -reciprocal encoding [57]. Then, the graph  $G = (\mathcal{V},\mathcal{E})$  can be defined as:
$$
\mathcal {V} = \left\{v _ {i} \mid v _ {i} = \left[ \boldsymbol {A} _ {i, 0}, \dots , \boldsymbol {A} _ {i, | \mathcal {B} _ {f} | - 1} \right], i \in \{0, \dots , | \mathcal {B} _ {f} | - 1 \} \right\}, \tag {4}
$$
$$
\mathcal {E} = \left\{e _ {i j} \mid e _ {i j} = \langle \boldsymbol {f} _ {i}, \boldsymbol {f} _ {j} \rangle , \boldsymbol {f} _ {j} \in \mathcal {N} \left(\boldsymbol {f} _ {i}, k _ {2}\right), i \in \{0, \dots , | \mathcal {B} _ {f} | - 1 \} \right\}, \tag {5}
$$
where  $\pmb{f}_j\in \mathcal{N}(\pmb {f}_i,k_2)$  denotes that  $\pmb{f}_{j}$  lies within the  $k_{2}$  nearest neighbors of  $\pmb{f}_i$  in  $\mathcal{B}_f$
Message Propagation. After building the contextual relational graph, the message propagation can further obtain high-order relations among  $k$ -reciprocal encoding acquired at the previous stage, similar to the query expansion technique [35].
$$
\boldsymbol {h} _ {i} ^ {(l + 1)} = \boldsymbol {h} _ {i} ^ {(l)} + \text {a g g r e g a t e} \left(\left\{e _ {i j} ^ {\alpha} \cdot \boldsymbol {h} _ {j} ^ {(l)} \mid e _ {i j} \in \mathcal {E} \right\}\right), \tag {6}
$$
where  $\alpha$  is a fixed value to realize the  $\alpha$ -weighted query expansion ( $\alpha$ -QE), and  $e_{ij} \in \mathcal{E}$  ensures that the most confident  $k_2$  nearest neighbors are selected for aggregation. Normally,  $k_2$  should be far less than  $k_1$ . For clarity, if we assume the number of layers for message propagation is  $L$ , the initial input for the message propagation stage,  $\pmb{h}_i^{(0)}$ , is derived from  $v_i$  from the previous stage. After propagation across all  $L$  layers, the output  $\pmb{h}_i^{(L)}$  becomes our desired contextually refined feature, represented as  $\pmb{h}_i^r$ .
Affinitive Neighborhood Retrieval. So far, the intra-batch data points have been mapped to a contextual metric space via a two-stage transformation. The underlying ranking statistics within these transformed data points should be more accurate and informative. Therefore, we reveal the hidden properties by explicitly retrieving the affinitive neighborhood using the refined node features, which we term as the contextually affinitive neighborhood discovery. Concretely, given the original features  $\left[\pmb{f}_0,\dots ,\pmb{f}_{|\mathcal{B}_f| - 1}\right]^T$  and the refined features  $\left[\pmb {h}_0^r,\dots ,\pmb {h}_{|\mathcal{B}_f| - 1}^r\right]^T$ , we define the affinitive neighborhood  $\mathcal{N}^a (i,k)$  as:
$$
\mathcal {N} ^ {a} (i, k) = \{j | \boldsymbol {h} _ {j} ^ {r} \in \mathcal {N} (\boldsymbol {h} _ {i} ^ {r}, k) \}. \tag {7}
$$
Then the group-aware discrimination can be reformulated as:
$$
\mathcal {L} _ {s i m} ^ {G A} = - \underset {\boldsymbol {x} _ {i} \in \mathcal {B}} {\mathbb {E}} [ \underset {j \in \mathcal {N} ^ {a} (i, k)} {\mathbb {E}} [ \langle q (\mathcal {F} _ {o} (t (\boldsymbol {x} _ {i})), \mathcal {F} _ {t} \left(t ^ {\prime} (\boldsymbol {x} _ {j})\right) \rangle ] ]. \tag {8}
$$
We remark that the contextually affinitive neighborhood discovery ensures more informative neighbors are involved for training at every iteration, and the superior contextual neighbor information lying between the non-differentiable transformed data points is synchronously propagated to the original features to improve the representation quality.
2.3 Progressive Boundary Filtering with Relaxation
In this section, we are focused on the inevitable noise existing in the neighborhood. Although the contextually affinitive neighborhood could offer more contextually relevant samples to boost training, there still exist some unreliable samples among the top-ranked neighbors. Such a phenomenon becomes more evident when the sampled data points lie on the decision boundary. As a consequence, the inclusion of contaminated neighbor information will cause noise accumulation, to the detriment of later training iterations. Therefore, we propose an online boundary sample filtering with progressive relaxation to alleviate the issue.
2.3.1 Online Boundary Sample Detection
In order to better assess the current clustering quality and detect the samples located at the boundary, we apply k-means after each training epoch, which is a common practice in former works [17, 38]. Instead of directly utilizing the consistency of pseudo labels among neighbors, we place more emphasis on the holistic cluster structure, which is to distinguish farther samples located at the cluster boundary with controlled degrees. Inspired by the silhouette score which measures the ratio between the mean intra-cluster distance and the mean nearest-cluster distance in an offline manner, we devise an approximate online version to serve a similar purpose.
Formally, given a sample  $\pmb{x}_i$  with pseudo label  $\pmb{\pi}(\pmb{x}_i)$ , and  $K$  cluster centroids  $\pmb{C} = [c_1, \dots, c_K]^T$  obtained from last epoch, we first approximate the mean intra-cluster distance  $d_I$  and nearest mean inter-cluster distance  $d_N$  by:
$$
d _ {i} ^ {I} = \left\| \boldsymbol {f} _ {i} - \boldsymbol {c} _ {\boldsymbol {\pi} \left(\boldsymbol {x} _ {i}\right)} \right\| _ {2}, \quad d _ {i} ^ {N} = \left\| \boldsymbol {f} _ {i} - \boldsymbol {c} _ {\boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {i}\right)} \right\| _ {2}, \tag {9}
$$
$$
\boldsymbol {\pi} ^ {\prime} \left(\boldsymbol {x} _ {i}\right) = \underset {k \in \{1, \dots , K \} \backslash \{\boldsymbol {\pi} \left(\boldsymbol {x} _ {i}\right) \}} {\arg \min } \| \boldsymbol {f} _ {i} - \boldsymbol {c} _ {k} \| _ {2}, \tag {10}
$$
And then the boundary ratio can be computed as:
$$
r _ {i} = 1 - \frac {d _ {i} ^ {N} - d _ {i} ^ {I}}{\operatorname* {m a x} \left(d _ {i} ^ {I} , d _ {i} ^ {N}\right)}. \tag {11}
$$
Normally, a large boundary ratio indicates the given sample is more likely to be located at the boundary, while a small ratio implies the sample is close to its cluster center, and it is safer to exploit non-boundary samples for training since they exhibit cleaner neighborhoods.
2.3.2 Progressive Relaxation
While boundary samples may possess inferior neighborhoods, it is barely true to ignore these samples during the entire clustering process. To make a balance, we progressively relax the restrictions of samples to promote neighborhood consistency. The motivations are two folds. First, we set a strict criteria to select clean samples at the initial stage to avoid the confirmation bias brought by noisy samples. Second, after the neighborhoods have been consistently refined using cleaner samples, the boundary samples can further improve the overall clustering quality without affecting the holistic structure, which can be validated in our ablation experiments.
Formally, given a batch  $\mathcal{B}$  sampled from the dataset, we adopt a linearly increased fraction ratio  $f r^{(t)}$  to dynamically control a filtering threshold  $\sigma^{(t)}$  which is used to filter out the hard boundary samples at epoch  $t$ .
$$
f r ^ {(t)} = f r ^ {\left(t _ {0}\right)} + \frac {1 - f r ^ {\left(t _ {0}\right)}}{t - t _ {0}} * \left(T - t _ {0}\right), \tag {12}
$$
Algorithm 1 The proposed algorithm CoNR
Require: Dataset  $\mathcal{D}$  , online network  $\theta$  , and target network  $\hat{\theta}$  1: Pre-train  $\theta ,\hat{\theta}$  on  $\mathcal{D}$ $\triangleright$  Eq. (1)   
2: while Clustering do   
3: Sample batch  $\mathcal{B}$  from  $\mathcal{D}$    
4: Transform batch features  $\mathcal{B}_f$  to  $\mathcal{H}_f$    
5: Retrieve contextually affinitive neighborhood  $\mathcal{N}^a$  for  $\mathcal{B}_f$  using  $\mathcal{H}_f$ $\triangleright$  Eq. (7)   
6: Progressively filter out boundary samples for clustering  $\triangleright$  Eq. (13)   
7: Update  $\theta$  with  $\mathcal{L}_{sim}^{GAF}$  and SGD optimizer  $\triangleright$  Eq. (14)   
8: Update  $\hat{\theta}$  with momentum moving average   
9: end while
where  $t$  is the current epoch,  $t_0$  is the epoch to start clustering, and  $T$  is the maximum number of epochs for group-aware concordance clustering. Then  $\sigma^{(t)}$  can be obtained by retrieving the largest ratio among the top  $|\mathcal{B}|*fr^{(t)}$  smallest boundary ratios in  $\{r_0,\dots ,r_{|\mathcal{B}| - 1}\}$ . According to the  $\sigma^{(t)}$ , we filter out these samples whose boundary ratios are too large to be involved in clustering at the current epoch  $t$ :
$$
\mathcal {B} ^ {(t)} = \left\{\boldsymbol {x} _ {i} \mid r _ {i} \leq \sigma^ {(t)}, i \in \{0, \dots , | \mathcal {B} | - 1 \} \right\}. \tag {13}
$$
Notably, our progressive boundary filtering strategy is essentially different from the self-labeling step that selects the most confident samples for fine-tuning in prior work [42, 29]. First, their selection strategy is completely based on prediction confidence or pseudo-label consistency, neglecting the intrinsic cluster structure to evaluate the difficulty of samples. Second, their self-labeling step is based on cross-entropy loss, which heavily relies on stronger augmentations to avoid overfitting. By contrast, our filtering strategy is proposed to search a robust neighborhood and avoid accumulation errors during the group-aware concordance, which does not require additional modules or augmentations.
2.4 The Overall Objective and Optimization
Based on Sec. 2.2 and Sec. 2.3, our final objective for group-aware concordance can be further formulated as:
$$
\mathcal {L} _ {s i m} ^ {G A F} = - \underset {\boldsymbol {x} _ {i} \in \left\{\mathcal {B} \backslash \mathcal {B} ^ {(t)} \right\}} {\mathbb {E}} [ \underset {j \in \mathcal {N} ^ {a} (i, k)} {\mathbb {E}} [ \langle q (\mathcal {F} _ {o} (t (\boldsymbol {x} _ {i})), \mathcal {F} _ {t} \left(t ^ {\prime} (\boldsymbol {x} _ {j})\right) \rangle ] ], \tag {14}
$$
where  $\boldsymbol{x}_i \in \{\mathcal{B} \backslash \mathcal{B}^{(t)}\}$  controls whether the samples should be selected as candidates to retrieve neighborhood, and  $j \in \mathcal{N}^{(a)}(i, k)$  determines whether the samples are contextually affinitive to the candidates. We do not exclude  $\mathcal{B}^{(t)}$  from  $\mathcal{N}^{(a)}(i, k)$  as our aim is to filter out unreliable candidates lying around the boundaries, which means that for each selected candidate, we take its ConAff neighborhood as a whole to preserve its original structure. The overall training pipeline is shown in Algorithm 1.