SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Youquan Liu $^{1,*}$  Lingdong Kong $^{1,2,*}$  Jun Cen $^{3}$  Runnan Chen $^{4}$  Wenwei Zhang $^{1,5}$  Liang Pan $^{5}$  Kai Chen $^{1}$  Ziwei Liu $^{5,\boxtimes}$
\(^{1}\)Shanghai AI Laboratory \(^{2}\)National University of Singapore \(^{3}\)The Hong Kong University of Science and Technology \(^{4}\)The University of Hong Kong \(^{5}\)S-Lab, Nanyang Technological University \(\{1iuyouquan, konglingdong, zhangwenwei, chenkai\} @pjlab.org.cn jcenaaa@connect.ust.hk \(\{1iang.pan, ziwei.liu\} @ntu.edu.sg\)
Recent advancements in vision foundation models (VFMs) have opened up new possibilities for versatile and efficient visual perception. In this work, we introduce Seal, a novel framework that harnesses VFMs for segmenting diverse automotive point cloud sequences. Seal exhibits three appealing properties: i) Scalability: VFMs are directly distilled into point clouds, obviating the need for annotations in either 2D or 3D during pretraining. ii) Consistency: Spatial and temporal relationships are enforced at both the camera-to-LiDAR and point-to-segment regularization stages, facilitating cross-modal representation learning. iii) Generalizability: Seal enables knowledge transfer in an off-the-shelf manner to downstream tasks involving diverse point clouds, including those from real/synthetic, low/high-resolution, large/small-scale, and clean/corrupted datasets. Extensive experiments conducted on eleven different point cloud datasets showcase the effectiveness and superiority of Seal. Notably, Seal achieves a remarkable  $45.0\%$  mIoU on nuScenes after linear probing, surpassing random initialization by  $36.9\%$  mIoU and outperforming prior arts by  $6.1\%$  mIoU. Moreover, Seal demonstrates significant performance gains over existing methods across 20 different few-shot fine-tuning tasks on all eleven tested point cloud datasets. The code is available at this link<sup>2</sup>.
Inspired by the achievements of large language models (LLMs) [87, 35, 113, 73, 18], a wave of vision foundation models (VFMs), such as SAM [50], X-Decoder [122], and SEEM [123], has emerged. These VFMs are revolutionizing the field of computer vision by facilitating the acquisition of pixel-level semantics with greater ease. However, limited studies have been conducted on developing VFMs for the 3D domain. To bridge this gap, it holds great promise to explore the adaptation or extension of existing 2D VFMs for 3D perception tasks.
As an important 3D perception task, accurately segmenting the surrounding points captured by onboard LiDAR sensors is crucial for ensuring the safe operation of autonomous vehicles [92, 94, 81]. However, existing point cloud segmentation models rely heavily on large annotated datasets for training, which poses challenges due to the labor-intensive nature of point cloud labeling [2, 95]. To address this issue, recent works have explored semi- [55, 59] or weakly-supervised [95, 60] approaches to alleviate the annotation burden. While these approaches show promise, the trained point segmentors tend to perform well only on data within their distribution, primarily due to
*Youquan and Lingdong contributed equally to this work. Ziwei serves as the corresponding author.
GitHub Repo: https://github.com/youquanl/Segment-Any-Point-Cloud.
Figure 1: The proposed Seal distills semantic awareness on cameras views from VFMs to the point cloud via superpixel-driven contrastive learning. [1st row] Semantic superpixels generated by SLIC [1] and recent VFMs [50, 122, 123], where each color represents one segment. [2nd row] Semantic superpoints grouped by projecting superpixels to 3D via camera-LiDAR correspondence. [3rd row] Visualizations of the linear probing results of our framework driven by SLIC and different VFMs.
significant configuration differences (such as beam number, camera angle, emit rate) among different sensors [101, 28, 92]. This limitation inevitably hampers the scalability of point cloud segmentation.
To address the aforementioned challenges, we aim to develop a framework that can learn informative features while addressing the following objectives:  $i$  ) Utilizing raw point clouds as input, thereby eliminating the need for semi or weak labels and reducing annotation costs. ii) Leveraging spatial and temporal cues inherent in driving scenes to enhance representation learning. iii) Ensuring generalizability to diverse downstream point clouds, beyond those used in the pretraining phase. Drawing inspiration from recent advancements in cross-modal representation learning [50, 122, 111, 123] and building upon the success of VFMs [50, 122, 111, 123], we propose a methodology that distills semantically-rich knowledge from VFMs to support self-supervised representation learning on challenging automotive point clouds. Our core idea is to leverage the 2D-3D correspondence in between LiDAR and camera sensors and construct high-quality contrastive samples for cross-modal representation learning. As shown in Fig. 1, VFMs are capable of generating semantic superpixel groups from the camera views $^{3}$  and provide off-the-shelf semantic coherence for distinct objects and backgrounds in the 3D scene. Such consistency can be distilled from 2D to 3D and further yields promising performance on downstream tasks, which we will revisit more formally in later sections.
Compared to previous framework [85, 66], our VFM-assisted contrastive learning has three notable advantages: i) The semantically-aware region partition mitigates the severe "self-conflict" problem in driving scene contrastive learning. ii) The high-quality contrastive samples gradually form a more coherent optimization landscape, yielding a faster convergence rate than previous configurations. iii) A huge reduction in the number of superpixels also extenuates the overhead during pretraining.
Since a perfect calibration between LiDAR and cameras is often hard to meet, we further design a superpoint temporal consistency regularization to mitigate the case that there are potential errors from the sensor synchronization aspect. This objective directly resorts to the accurate geometric information from the point cloud and thus improves the overall resilience and reliability. As will be shown in the following sections, our framework (dubbed Seal) is capable of segmentation any point cloud sequences across a wide range of downstream task on different automotive point cloud datasets.
To summarize, this work has the following key contributions:
- To the best of our knowledge, this study represents the first attempt at utilizing 2D vision foundation models for self-supervised representation learning on large-scale 3D point clouds.
3We show the front view for simplicity; more common setups have surrounding views from multiple cameras.
- We introduce Seal, a scalable, consistent, and generalizable framework designed to capture semantic-aware spatial and temporal consistency, enabling the extraction of informative features from automotive point cloud sequences.
- Our approach demonstrates clear superiority over previous state-of-the-art (SoTA) methods in both linear probing and fine-tuning for downstream tasks across 11 different point cloud datasets with diverse data configurations.
Vision Foundation Models. Recent excitement about building powerful visual perception systems based on massive amounts of training data [80, 50] or advanced self-supervised learning techniques [8, 74] is revolutionizing the community. The segment anything model (SAM) [50] sparks a new trend of general-purpose image segmentation which exhibits promising zero-shot transfer capability on diverse downstream tasks. Concurrent works to SAM, such as X-Decoder [122], OpenSeeD [111], SegGPT [99], and SEEM [123], also shed lights on the direct use of VFMs for handling different kinds of image-related tasks. In this work, we extend this aspect by exploring the potential of VFMs for point cloud segmentation. We design a new framework taking into consideration the off-the-shelf semantic awareness of VFMs to construct better spatial and temporal cues for representation learning.
Point Cloud Segmentation. Densely perceiving the 3D surroundings is important for autonomous vehicles [94, 41]. Various point cloud segmenters have been proposed, including methods based on raw points [42, 90, 43, 114, 78], range view [69, 106, 93, 118, 20, 16, 52], bird's eye view [115, 119, 9], voxels [19, 120, 37, 36], and multi-view fusion [62, 89, 107, 121, 17, 79]. Despite the promising results achieved, existing 3D segmentation models rely on large sets of annotated data for training, which hinders the scalability [28]. Recent efforts seek semi [55, 56, 59], weak [95, 40, 86, 63, 60], and active [64, 44, 117] supervisions or domain adaptation techniques [48, 47, 54, 76, 82, 68] to ease the annotation cost. In this work, we resort to self-supervised learning by distilling foundation models using camera-to-LiDAR associations, where no annotation is required during the pretraining stage.
3D Representation Learning. Stemmed from the image vision community, most 3D self-supervised learning methods focused on object-centric point clouds [84, 77, 83, 14, 97, 45, 110, 29, 91] or indoor scenes [38, 15, 22, 108, 10, 39, 57, 58, 109] by either pretext task learning [21, 70, 71, 112, 30], contrastive learning [11, 32, 12, 34, 27, 13, 46, 98, 104] or mask modeling [105, 31, 24, 61], where the scale and diversity are much lower than the outdoor driving scenes [67, 6]. PointContrast [103], DepthContrast [116], and SegContrast [72] are prior attempts aimed to establish contrastive objectives on point clouds. Recently, Sautier et al. [85] proposed the first 2D-to-3D representation distillation method called SLidR for cross-modal self-supervised learning on large-scale point clouds and exhibits promising performance. The follow-up work [66] further improves this pipeline with a semantically tolerant contrastive constraint and a class-balancing loss. Our framework also stems from the SLidR paradigm. Differently, we propose to leverage VFMs to establish the cross-modal contrastive objective which better tackles this challenging representation learning task. We also design a superpoint temporal consistency regularization to further enhance feature learning.
3 Seal: A Scalable, Consistent, and Generalizable Framework
In this section, we first revisit the 2D-to-3D representation distillation [85] as preliminaries (Sec. 3.1). We then elaborate on the technical details of our framework, which include the VFM-assisted spatial contrastive learning (Sec. 3.2) and the superpoint temporal consistency regularization (Sec. 3.3).
Given a point cloud  $\mathcal{P}^t = \{\mathbf{p}_i^t, \mathbf{e}_i^t | i = 1, \dots, N\}$  consists of  $N$  points collected by a LiDAR acquisition at timestamp  $t$ , where  $\mathbf{p}_i \in \mathbb{R}^3$  denotes the point coordinates and  $\mathbf{e}_i \in \mathbb{R}^L$  is the feature embedding (intensity, elongation, etc.). our goal is to transfer the knowledge from an image set  $\mathcal{I}^t = \{\{\mathbf{I}_1^t, \dots, \mathbf{I}_j^t\} | j = 1, \dots, V\}$  captured by  $V$  synchronized cameras at  $t$  to point cloud  $\mathcal{P}^t$ , where  $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}$  is a single image with height  $H$  and width  $W$ . Prior works [85, 66] achieve this goal by first aggregating image regions that are similar in the RGB space into a superpixel set  $\Phi_S = \{\{\mathbf{s}_1, \dots, \mathbf{s}_m\} | m = 1, \dots, M\}$ , via the unsupervised SLIC [1] algorithm. The corresponding
Figure 2: Overview of the Seal framework. We generate, for each  $\{\mathrm{LiDAR},\mathrm{camera}\}$  pair  $\{\mathcal{P}^t,\mathcal{I}^t\}$  at timestamp  $t$  and another LiDAR frame  $\mathcal{P}^{t + n}$  at timestamp  $t + n$ , the semantic superpixel and superpoint by vision foundation models (VFMs). Two pertaining objectives are then formed, including spatial contrastive learning between paired LiDAR and camera features (Sec. 3.2) and temporal consistency regularization between point segments at different timestamps (Sec. 3.3).
superpoint set  $\Phi_{\mathcal{O}} = \{\{\mathbf{o}_1,\dots,\mathbf{o}_m\} |m = 1,\dots,M\}$  can be obtained by leveraging known sensor calibration parameters to establish correspondence between the points and image pixels. Specifically, since the LiDAR and cameras usually operate at different frequencies [7, 26], we transform each LiDAR point cloud  $\mathbf{p}_i = (x_i,y_i,z_i)$  at timestamp  $t_l$  to the pixel  $\hat{\mathbf{p}}_i = (u_i,v_i)\in \mathbb{R}^2$  in the image plane at timestamp  $t_c$  via the coordinate transformation. The point-to-pixel mapping is as follows:
$$
\left[ u _ {i}, v _ {i}, 1 \right] ^ {\mathbf {T}} = \frac {1}{z _ {i}} \times \Gamma_ {K} \times \Gamma_ {\text {c a m e r a} \leftarrow \mathrm {e g o} _ {\mathrm {t} _ {c}}} \times \Gamma_ {\mathrm {e g o} _ {\mathrm {t} _ {c}} \leftarrow \text {g l o b a l}} \times \Gamma_ {\text {g l o b a l} \leftarrow \mathrm {e g o} _ {\mathrm {t} _ {l}}} \times \Gamma_ {\mathrm {e g o} _ {\mathrm {t} _ {l}} \leftarrow \text {l i d a r}} \times \left[ x _ {i}, y _ {i}, z _ {i}, 1 \right] ^ {\mathbf {T}}, \tag {1}
$$
where symbol  $\Gamma_K$  denotes the camera intrinsic matrix.  $\Gamma_{\mathrm{camera}} \leftarrow \mathrm{ego}_{t_c}$ ,  $\Gamma_{\mathrm{ego}_{t_c}} \leftarrow \mathrm{global}$ ,  $\Gamma_{\mathrm{global}} \leftarrow \mathrm{ego}_{t_l}$  and  $\Gamma_{\mathrm{ego}_{t_l}} \leftarrow \mathrm{lidar}$  are the extrinsic matrices for the transformations of ego to camera at  $t_c$ , global to ego at  $t_c$ , ego to global at  $t_l$ , and LiDAR to ego at  $t_l$ , respectively.
3.2 Semantic Superpixel Spatial Consistency
Superpixel Generation. Prior works resort to SLIC [1] to group visually similar regions in the image into superpixels. This method, however, tends to over-segment semantically coherent areas (as shown in Fig. 1) and inevitably leads to several difficulties for contrastive learning. One of the main impediments is the so-called "self-conflict", where superpixels belonging to the same semantics become negative samples [96]. Although [66] proposed a semantically-tolerant loss to ease this problem, the lack of high-level semantic understanding still intensifies the implicit hardness-aware property of the contrastive loss. We tackle this challenge by generating semantic superpixels with VFMs. As shown in Fig. 1 and Fig. 4, these VFMs provide semantically-rich superpixels and yield much better representation learning effects in-between near and far points in the LiDAR point cloud.
VFM-Assisted Contrastive Learning. Let  $F_{\theta_p}:\mathbb{R}^{N\times (3 + L)}\to \mathbb{R}^{N\times C}$  be a 3D encoder with trainable parameters  $\theta_{p}$ , that takes a LiDAR point cloud as input and outputs a  $C$ -dimensional per-point feature. Let  $G_{\theta_i}:\mathbb{R}^{H\times W\times 3}\to \mathbb{R}^{\frac{H}{s}\times \frac{W}{s}\times C}$  be an image encoder with parameters  $\theta_{i}$  which is initialized from a set of 2D self-supervised pretrained parameters. The goal of our VFM-assisted contrastive learning is to transfer the knowledge from the pretrained 2D network to the 3D network via contrastive loss at the semantic superpixel level. To compute this VFM-assisted contrastive loss, we first build trainable projection heads  $H_{\omega_p}$  and  $H_{\omega_i}$  which map the 3D point features and 2D image features into the same  $D$ -dimensional embedding space. The point projection head  $H_{\omega_p}:\mathbb{R}^{N\times C}\to \mathbb{R}^{N\times D}$  is a linear layer with  $\ell_2$ -normalization. The image projection head
$H_{\omega_i}:\mathbb{R}^{\frac{H}{s}\times \frac{W}{s}\times C}\to \mathbb{R}^{\frac{H}{s}\times \frac{W}{s}\times D}$  is a convolution layer with a kernel size of 1, followed by a fixed bilinear interpolation layer with a ratio of 4 in the spatial dimension, and outputs with  $\ell_2$  -normalization.
We distill the knowledge from the 2D network into the 3D network which is in favor of a solution where a semantic superpoint feature has a strong correlation with its corresponding semantic superpixel feature than any other features. Concretely, the superpixels  $\Phi_S$  are used to group pixel embedding and point embedding features. Then, an average pooling function is applied to each grouped point and pixel embedding features, to extract the superpixel embedding features  $\mathbf{Q} \in \mathbb{R}^{M \times D}$  and superpoint embedding features  $\mathbf{K} \in \mathbb{R}^{M \times D}$ . The VFM-assisted contrastive loss is formulated as:
$$
\mathcal {L} ^ {v f m} = \mathcal {L} (\mathbf {Q}, \mathbf {K}) = - \frac {1}{M} \sum_ {i = 1} ^ {M} \log \left[ \frac {e ^ {(\langle \mathbf {q} _ {i} , \mathbf {k} _ {i} \rangle / \tau)}}{\sum_ {j \neq i} e ^ {(\langle \mathbf {q} _ {i} , \mathbf {k} _ {j} \rangle / \tau)} + e ^ {(\langle \mathbf {q} _ {i} , \mathbf {k} _ {i} \rangle / \tau)}} \right], \tag {2}
$$
where  $\langle \mathbf{q}_i,\mathbf{k}_j\rangle$  denotes the scalar product between superpoint embedding features and superpixel embedding features to measure the similarity.  $\tau$  is the temperature term.
Role in Our Framework. Our VFM-assisted contrastive objective exhibits superiority over previous methods from three aspects: i) The semantically-rich superpixels provided by VFMs mitigate the "self-conflict" problem in existing approaches. ii) As we will show in the following sections, the high-quality contrastive samples from VFMs gradually form a much more coherent optimization landscape and yield a faster convergence rate than the unsupervised superpixel generation method. iii) Using superpixels generated by VFMs also helps our framework run faster than previous works, since the embedding length of  $\mathbf{Q}$  and  $\mathbf{K}$  has been reduced from a few hundred (SLIC [1]) to dozens (ours).
3.3 Semantic Superpoint Temporal Consistency
The assumption of having perfectly synchronized LiDAR and camera data might become too ideal and cannot always be fulfilled in actual deployment, which limits the scalability. In this work, we resort to accurate geometric information from point clouds to further relieve this constraint.
Implicit Geometric Clustering. To group coarse instance segments in a LiDAR frame, we first partition non-ground plane points  $\mathcal{G}^t$  by eliminating the ground plane of a point cloud  $\mathcal{P}^t$  at timestamp  $t$  in an unsupervised manner via
RANSAC [25]. Then, we group  $\mathcal{G}^t$  to yield a set of  $M_{k}$  segments  $\mathcal{K}^t = \{\mathcal{K}_1^t,\dots,\mathcal{K}_{M_k}^t\}$  with the help of HDBSCAN [23]. To map different segment views at different timestamps, we transform those LiDAR frames across different timestamps to the global frame and aggregate them with concatenations. The aggregated point cloud is denoted as  $\widetilde{\mathcal{P}} = \{\widetilde{\mathcal{P}}^t,\dots,\widetilde{\mathcal{P}}^{t + n}\}$ . Similarly, we generate non-ground plane  $\widetilde{\mathcal{G}} = \{\widetilde{\mathcal{G}}^t,\dots,\widetilde{\mathcal{G}}^{t + n}\}$  from  $\widetilde{\mathcal{P}}$  via RANSAC [25]. In the same manner as the single scan, we group  $\widetilde{\mathcal{G}}$  to obtain  $M_{k}$  segments  $\widetilde{\mathcal{K}} = \{\widetilde{\mathcal{K}}_1,\dots,\widetilde{\mathcal{K}}_{M_k}\}$ . To generate the segment masks for all  $n + 1$  scans at  $n$  consecutive timestamps, i.e.,  $\widetilde{\mathcal{K}} = \{\widetilde{\mathcal{K}}^t,\dots,\widetilde{\mathcal{K}}^{t + n}\}$ , we maintain the point index mapping from the aggregated point cloud  $\widetilde{\mathcal{P}}$  to the  $n + 1$  individual scans.
Superpoint Temporal Consistency. We leverage the clustered segments to compute the temporal consistency loss among related semantic superpoints. Here we assume  $n = 1$  (i.e. the next frame) without loss of generalizability. Specifically, given a sampled temporal pair  $\widetilde{\mathcal{P}}^t$  and  $\widetilde{\mathcal{P}}^{t + 1}$  and their corresponding segments  $\widetilde{\mathcal{K}}^t$  and  $\widetilde{\mathcal{K}}^{t + 1}$ , we compute the point-wise features  $\hat{\mathcal{F}}^t\in \mathbb{R}^{N\times D}$  and  $\hat{\mathcal{F}}^{t + 1}\in \mathbb{R}^{N\times D}$  from the point projection head  $H_{\omega_p}$ . As for the target embedding, we split the point features  $\hat{\mathcal{F}}^t$  and  $\hat{\mathcal{F}}^{t + 1}$  into  $M_{k}$  groups by segments  $\widetilde{\mathcal{K}}^t$  and  $\widetilde{\mathcal{K}}^{t + 1}$ . Then, we apply an average pooling operation on  $\hat{\mathcal{F}}^{t + 1}$  to get  $M_{k}$  target mean feature vectors  $\mathcal{F}^{t + 1} = \{\mathcal{F}_1^{t + 1},\mathcal{F}_2^{t + 1},\dots,\mathcal{F}_{M_k}^{t + 1}\}$ , where  $\mathcal{F}_{M_k}^{t + 1}\in \mathbb{R}^{1\times D}$ . Let the split point feature  $\hat{\mathcal{F}}^t$  be  $\mathcal{F}^t = \{\mathcal{F}_1^t,\mathcal{F}_2^t,\dots,\mathcal{F}_{M_k}^t\}$ , where  $\mathcal{F}_{M_k}^t\in \mathbb{R}^{k\times D}$  and
Figure 3: The positive feature correspondences in the contrastive learning objective in our contrastive learning framework. The circles and triangles represent the instance-level and the point-level features, respectively.
Figure 4: The cosine similarity between a query point (denoted as the red dot) and the feature learned with SLIC [1] and different VFMs [50, 122, 111, 123]. The queried semantic classes from top to bottom examples are: "car", "manmade", and "truck". The color goes from violet to yellow denoting low and high similarity scores, respectively. Best viewed in color.
$k$  is the number of points in the corresponding segment. We compute the temporal consistency loss  $\mathcal{L}^{t\rightarrow t + 1}$  to minimize the differences between the point features in the current frame (timestamp  $t$ ) and the corresponding segment mean features from the next frame (timestamp  $t + 1$ ) as follows:
$$
\mathcal {L} ^ {t \rightarrow t + 1} = - \frac {1}{M _ {k}} \sum_ {i = 1} ^ {M _ {k}} \log \left[ \frac {e ^ {\left(\left\langle \mathbf {f} _ {i} ^ {t} , \mathbf {f} _ {i} ^ {t + 1} \right\rangle / \tau\right)}}{\sum_ {j \neq i} e ^ {\left(\left\langle \mathbf {f} _ {i} ^ {t} , \mathbf {f} _ {j} ^ {t + 1} \right\rangle / \tau\right)} + e ^ {\left(\left\langle \mathbf {f} _ {i} ^ {t} , \mathbf {f} _ {i} ^ {t + 1} \right\rangle / \tau\right)}} \right]. \tag {3}
$$
Since the target embedding for all points within a segment in the current frame serves as the mean segment representation from the next frame, this loss will force points from a segment to converge to a mean representation while separating from other segments, implicitly clustering together points from the same instance. Fig. 3 provides the positive feature correspondence in our contrastive learning framework. Furthermore, we swap  $\hat{\mathcal{F}}^t$  when generating the target mean embedding features to form a symmetric representation. In this way, the correspondence is encouraged from both  $t\to t + 1$  and  $t + 1\rightarrow t$ , which leads to the following optimization objective:  $\mathcal{L}^{tmp} = \mathcal{L}^{t\to t + 1} + \mathcal{L}^{t + 1\to t}$ .
Point to Segment Regularization. To pull close the LiDAR points belonging to the same instance at timestamp  $t$ , we minimize the distance between the point feature  $\mathcal{F}^t$  and the corresponding mean cluster feature  $\mathcal{C}^t$ . To implement this, we leverage a max-pooling function to pool  $\mathcal{F}^t$  according to the segments to obtain  $\mathcal{C}^t = \{\mathcal{C}_1^t, \mathcal{C}_2^t, \dots, \mathcal{C}_{M_k}^t\}$ , where  $\mathcal{C}_{M_k}^t \in \mathbb{R}^{1 \times D}$ . The point-to-segment regularization is thus achieved via the following loss function:
$$
\mathcal {L} ^ {p 2 s} = - \frac {1}{M _ {k} N _ {k}} \sum_ {i = 1} ^ {M _ {k}} \sum_ {a = 1} ^ {N _ {k}} \log \left[ \frac {e ^ {\left(\left\langle \mathbf {c} _ {i} ^ {t} , \mathbf {f} _ {i , a} ^ {t} \right\rangle / \tau\right)}}{\sum_ {j \neq i} e ^ {\left(\left\langle \mathbf {c} _ {i} ^ {t} , \mathbf {f} _ {j , a} ^ {t} \right\rangle / \tau\right)} + e ^ {\left(\left\langle \mathbf {c} _ {i} ^ {t} , \mathbf {f} _ {i , a} ^ {t} \right\rangle / \tau\right)}} \right], \tag {4}
$$