Reviewerly/exHarmony huggingface dataset:
A Feature Refinement Patch Embedding-Based Recognition Method for Printed Tibetan Cursive Script

Recognition of Tibetan cursive scripts has important applications in the field of automated Tibetan office software and ancient document conservation. However, there are few studies on recognition of Tibetan cursive scripts. This paper proposes a printed Tibetan cursive script recognition method based on feature refinement patch embedding. Firstly, the feature refinement patch embedding module (FRPE) serializes the line text image of feature sequences. Secondly, a global modeling of feature vectors is carried out by using a single transformer encoder. Finally, the output of the recognition result is decoded by using a fully connected layer. Experimental results show that, compared with the baseline model, the proposed method improves the accuracy by 9.52% on the dataset CSTPD, a database containing six Tibetan cursive fonts. Moreover, it achieves an average accuracy rate of 92.5% on the dataset CSTPD. Similarly, it also works better than the baseline model on Tibetan text recognition synthetic data for natural scene images.
QEA-Net: Quantum-Effects-based Attention Networks

In the past decade, the attention mechanism has played an increasingly important role in computer vision. Such an attention mechanism can be regarded as a dynamic weight adjustment process based on features of the input image. In this paper, we propose Quantum-Effects-based Attention Networks (QEA-Net), the simple yet effective attention networks, they can be integrated into many network architectures seamlessly. QEA-Net uses quantum effects between two identical particles to enhance the global channel information representation of the attention module. Our method could consistently outperform the SENet, with a lower number of parameters and computational cost. We evaluate QEA-Net through experiments on ImageNet-1K and compare it with state-of-the-art counterparts. We also demonstrated the effect of QEA-Net in combination with pre-trained networks on small downstream transfer learning tasks.
Classifier Decoupled Training for Black-Box Unsupervised Domain Adaptation

Black-box unsupervised domain adaptation ( $$\textrm{B}^2 \textrm{UDA}$$ ) is a challenging task in unsupervised domain adaptation, where the source model is treated as a black box and only its output is accessible. Previous works have treated the source models as a pseudo-labeling tool and formulated $$\textrm{B}^2 \textrm{UDA}$$ as a noisy labeled learning (LNL) problem. However, they have ignored the gap between the “shift noise” caused by the domain shift and the hypothesis noise in LNL. To alleviate the negative impact of shift noise on $$\textrm{B}^2 \textrm{UDA}$$ , we propose a novel framework called Classifier Decoupling Training (CDT), which introduces two additional classifiers to assist model training with a new label-confidence sampling. First, we introduce a self-training classifier to learn robust feature representation from the low-confidence samples, which is discarded during testing, and the final classifier is only trained with a few high-confidence samples. This step decouples the training of high-confidence and low-confidence samples to mitigate the impact of noise labels on the final classifier while avoiding overfitting to the few confident samples. Second, an adversarial classifier optimizes the feature distribution of low-confidence samples to be biased toward high-confidence samples through adversarial training, which greatly reduces intra-class variation. Third, we further propose a novel ETP-entropy Sampling (E $$^2$$ S) to collect class-balanced high-confidence samples, which leverages the early-time training phenomenon into LNL. Extensive experiments on several benchmarks show that the proposed CDT achieves $$88.2\%$$ , $$71.6\%$$ , and $$81.3\%$$ accuracies on Office-31, Office-Home, and VisDA-17, respectively, which outperforms state-of-the-art methods.
Boosting Out-of-Distribution Detection with Sample Weighting

To enhance the reliability of machine learning models in the open world, there is considerable interest in detecting out-of-distribution (OOD) inputs that lie far away from the training distribution. Existing distance-based methods have shown promising performance in OOD detection. These methods generally follow a common approach of capturing the distances between training samples and each test sample in the feature space. This can be understood as encoding the distance information to assign sample weights, which are then used to calculate a weighted distance score to determine if the input is OOD. However, these methods often adopt a coarse-grained weighting approach, where only a small fraction of the training samples are considered and given weights. Consequently, they fail to fully leverage the complete distance information of the training data, leading to occasional difficulties in effectively distinguishing OOD samples. In this paper, we propose a novel approach to encode the complete distance information of the training data by assigning a weight to each sample based on its distance from the test sample, with the weights decaying as the distance increases. Furthermore, we introduce a weighted distance-based method for OOD detection. We demonstrate the superiority of our method over most existing supervised OOD detectors. Particularly, on a hard OOD detection task involving CIFAR-100 vs. CIFAR-10, our method achieves a reduction of $$1.82\%$$ in the average FPR95 compared to the current best method KNN.
Adaptable Conservative Q-Learning for Offline Reinforcement Learning

The Out-of-Distribution (OOD) issue presents a considerable obstacle in offline reinforcement learning. Although current approaches strive to conservatively estimate the Q-values of OOD actions, their excessive conservatism under constant constraints may adversely affect model learning throughout the policy learning procedure. Moreover, the diverse task distributions across various environments and behaviors call for tailored solutions. To tackle these challenges, we propose the Adaptable Conservative Q-Learning (ACQ) method, which capitalizes on the Q-value’s distribution for each fixed dataset to devise a highly generalizable metric that strikes a balance between the conservative constraint and the training objective. Experimental outcomes reveal that ACQ not only holds its own against a variety of offline RL algorithms but also significantly improves the performance of CQL on most D4RL MuJoCo locomotion tasks in terms of normalized return.
Learning Scene Graph for Better Cross-Domain Image Captioning

The current image captioning (IC) methods achieve good results within a single domain primarily due to training on a large amount of annotated data. However, the performance of single-domain image captioning methods suffers when extended to new domains. To address this, we propose a cross-domain image captioning framework, called SGCDIC, which achieves cross-domain generalization of image captioning models by simultaneously optimizing two coupled tasks, i.e., image captioning and text-to-image synthesis (TIS). Specifically, we propose a scene-graph-based approach SGAT for image captioning tasks. The image synthesis task employs a GAN variant (DFGAN) to synthesize plausible images based on the generated text descriptions by SGAT. We compare the generated images with the real images to enhance the image captioning performance in new domains. We conduct extensive experiments to evaluate the performance of SGCDIC by using the MSCOCO as the source domain data, and using Flickr30k and Oxford-102 as the new domain data. Sufficient comparative experiments and ablation studies demonstrate that SGCDIC achieves substantially better performance than the strong competitors for the cross-domain image captioning task.
Enhancing Rule Learning on Knowledge Graphs Through Joint Ontology and Instance Guidance

Rule learning is a machine learning method that extracts implicit rules and patterns from data, enabling symbol-based reasoning in artificial intelligence. Unlike data-driven approaches such as deep learning, using rules for inference allows for interpretability. Many studies have attempted to automatically learn first-order rules from knowledge graphs. However, existing methods lack attention to the hierarchical information between ontology and instance and require a separate rule evaluation stage for rule filtering. To address these issues, this paper proposes a Ontology and Instance Guided Rule Learning (OIRL) approach to enhance rule learning on knowledge graphs. Our method treats rules as sequences composed of relations and utilizes semantic information from both ontology and instances to guide path generation, ensuring that paths contain as much pattern information as possible. We also develop an end-to-end rule generator that directly infers rule heads and incorporates an attention mechanism to output confidence scores, eliminating the need for rule instance-based rule evaluation. We evaluate OIRL using the knowledge graph completion task, and experimental results demonstrate its superiority over existing rule learning methods, confirming the effectiveness of the mined rules.
Unsupervised Domain Adaptation for Optical Flow Estimation

In recent years, we have witnessed significant breakthroughs of optical flow estimation with the thriving of deep learning. The performance of the unsupervised method is unsatisfactory due to it is lack of effective supervision. The supervised approaches typically assume that the training and test data are drawn from the same distribution, which is not always held in practice. Such a domain shift problem are common exists in optical flow estimation and makes a significant performance drop. In this work, we address these challenge scenarios and aim to improve the model generalization ability of the cross-domain optical flow estimation model. Thus we propose a novel framework to tackle the domain shift problem in optical flow estimation. To be specific, we first design a domain adaptive autoencoder to transform the source domain and the target domain image into a common intermediate domain. We align the distribution between the source and target domain in the latent space by a discriminator. And the optical flow estimation module adopts the images in the intermediate domain to predict the optical flow. Our model can be trained in an end-to-end manner and can be a plug and play module to the existing optical flow estimation model. We conduct extensive experiments on the domain adaptation scenarios including Virtual KITTI to KITTI and FlyingThing3D to MPI-Sintel, the experimental results show the effectiveness of our proposed method.
CasFormer: Cascaded Transformer Based on Dynamic Voxel Pyramid for 3D Object Detection from Point Clouds

Recently, Transformers have been widely applied in 3-D object detection to model global contextual relationships in point cloud collections or for proposal refinement. However, the structural information in 3-D point clouds, especially to the distant and small objects is often incomplete, leading to difficulties in accurate detection using these methods. To address this issue, we propose a Cascaded Transformer based on Dynamic Voxel Pyramid (called CasFormer) for 3-D object detection from LiDAR point clouds. Specifically, we dynamically spread relevant features from the voxel pyramid based on the sparsity of each region of interest (RoI), capturing more rich semantic information for structurally incomplete objects. Furthermore, a cross-stage attention mechanism is employed to cascade the refined results of the Transformer in stage by stage, as well as to improve the training convergence of transformer. Extensive experiments demonstrate that our CasFormer achieves progressive performance in KITTI Dataset and Waymo Open Dataset. Compared to CT3D, our method outperforms it by 1.12% and 1.27% in the moderate and hard levels of car detection, respectively, on the KITTI online 3-D object detection leaderboard.
Focus the Overlapping Problem on Few-Shot Object Detection via Multiple Predictions

The overlapping problem is one of the main challenges of few-shot object detection (FSOD). It refers to the situation where some parts of the target are obscured by other objects or occluders. Since the support set is insufficient to provide enough samples with overlapping objects, it is difficult to get the correct bounding boxes due to the missing information. In this paper, we aim to detect highly overlapping instances in few-shot scenes and present a proposal-based method by combining both basic training and fine-tuning stages. Specifically, we predict a set of bounding boxes instead of a single bounding box for each proposal after proposal generation. Then, we introduce a new NMS strategy to prevent the erroneous removal of our desired bounding box by traditional NMS. Simultaneously, we introduce a new loss to distinguish the obtained bounding boxes to avoid the convergence of bounding boxes during the final training process. While benchmarking on MS-COCO and Pascal VOC, our method is confirmed to be efficient and obtains good generalizability. Our proposed method served as a good stimulus for future study in the field of FSOD.
Generalizable Neural Radiance Field with Hierarchical Geometry Constraint

The emergence of the generalized neural radiance field models has greatly expanded the applicability of novel view synthesis task. However, existing methods primarily rely on 2D features obtained from neighboring views to assist the rendering, which often leads to degradation in unseen viewpoints. Some approaches introduce cost volumes in the model to provide geometric priors, but such coarse geometric information fails to constrain the model effectively, resulting in rendered images with abundant artifacts. In this work, we propose to provide the model with hierarchical geometry constraints to achieve better rendering results. We introduce cascaded MVSNet to provide hierarchical scene structural features for inferring the underlying scene geometry. It constrains the model’s rendering under unseen viewpoints. Also, the hierarchical features provide fine representations that helps the reconstruction of details. Additionally, we utilize the hierarchical depth maps generated by cascade MVSNet to constrain the sampling process, ensuring that the sampled points are concentrated near the scene surface. Such sampling strategy filtering out numerous useless sampling points and improving both the sampling efficiency and rendering quality. Unlike previous generalized methods, we employ a new weight function proposed by Neus to eliminate inherent biases in the density field, disentangling the prediction of color and density with a signed distance field. Extensive experiments have demonstrated that our proposed method significantly improves the rendering quality and outperforms previous methods.
Inter-image Discrepancy Knowledge Distillation for Semantic Segmentation

As a typical dense prediction task, semantic segmentation remains challenging in industrial automation, since it is non-trivial to achieve a good tradeoff between the performance and the efficiency. Meanwhile, knowledge distillation has been applied to reduce the computational cost in semantic segmentation task. However, existing knowledge distillation methods for semantic segmentation mainly mimic the teachers’ behaviour using the well-designed knowledge variants from a single image, failing to explore discrepancy knowledge between different images. Considering that the large pre-trained teacher network usually tends to form a more robust discrepancy space than the small student, we propose a new inter-image discrepancy knowledge distillation method (IIDKD) for semantic segmentation. Extensive experiments are conducted on two popular semantic segmentation datasets, where the experimental results show the efficiency and effectiveness of distilling inter-image discrepancy knowledge.
Progressive Frequency-Aware Network for Laparoscopic Image Desmoking

Laparoscopic surgery offers minimally invasive procedures with better patient outcomes, but smoke presence challenges visibility and safety. Existing learning-based methods demand large datasets and high computational resources. We propose the Progressive Frequency-Aware Network (PFAN), a lightweight GAN framework for laparoscopic image desmoking, combining the strengths of CNN and Transformer for progressive information extraction in the frequency domain. PFAN features CNN-based Multi-scale Bottleneck-Inverting (MBI) Blocks for capturing local high-frequency information and Locally-Enhanced Axial Attention Transformers (LAT) for efficiently handling global low-frequency information. PFAN efficiently desmokes laparoscopic images even with limited training data. Our method outperforms state-of-the-art approaches in PSNR, SSIM, CIEDE2000, and visual quality on the Cholec80 dataset and retains only 629K parameters. Our code and models are made publicly available at: https://github.com/jlzcode/PFAN .
Multimodal Causal Relations Enhanced CLIP for Image-to-Text Retrieval

Traditional image-to-text retrieval models learn joint representations by aligning multimodal features, typically learning the weak correlation between image and text data which can introduce noise during modality alignment. To solve this problem, we propose a Multimodal Causal CLIP (MMC-CLIP) network that integrates causal semantic relationships into CLIP for image-to-text retrieval task. Firstly, we employ the Multimodal Causal Discovery (MCD) method, which models the causal relationships of causal variables in both image and text data to construct the multimodal causal graph. Subsequently, we seamlessly integrate the causal nodes extracted from the multimodal causal graph as learnable prompts within the CLIP model, giving rise to the novel framework of Multimodal Causal CLIP. By integrating causal semantics into CLIP, MMC-CLIP effectively enhances the correlation between causal variables in image and text modalities, leading to an improved alignment capability for multimodal image-text data. We demonstrate the superiority and generalization of the proposed method by outperforming all strong baselines in image-to-text retrieval task on the Flickr30K and MSCOCO datasets.
Co-attention Guided Local-Global Feature Fusion for Aspect-Level Multimodal Sentiment Analysis

Aspect-level multimodal sentiment analysis is a target oriented fine-grained sentiment analysis task aimed at determining the sentiment polarity of a given aspect of a sentence in conjunction with relevant multimodal data. Multimodal alignment and fusion remains a challenge for this task, and this paper proposes to solve this issue by considering the inter-modal local interactions. Therefore, a co-attention guided local-global feature fusion (CLGFF) method is proposed. The CLGFF method mines both aspect-guided global multimodal features and local fine-grained alignment between multimodalities, and then fuses them together for better exploitation of the global-local semantic correlation. A large number of experiments are carried out on two aspect-level multimodal sentiment datasets. A series of methods are compared from the experiments, and the results show that the proposed CLGFF method can better capture the local semantic correlation within the modality and the fine-grained consistency between different modalities, thereby improving the performance of aspect-level multimodal sentiment analysis.
An Internal-External Constrained Distillation Framework for Continual Semantic Segmentation

Deep neural networks have a notorious catastrophic forgetting problem when training serialized tasks on image semantic segmentation. It refers to the phenomenon of forgetting previously learned knowledge due to the plasticity-stability dilemma and background shift in the segmentation task. Continual Semantic Segmentation (CSS) comes into being to handle this challenge. Previous distillation-based methods only consider the knowledge of the features at the same level but neglect the relationship between different levels. To alleviate this problem, in this paper, we propose a mixed distillation framework called Internal-external Constrained Distillation (ICD), which includes multi-information-based internal feature distillation and attention-based external feature distillation. Specifically, we utilize the statistical information of features to perform internal distillation between the old model and the new model, which effectively avoids interference at the same scale. Furthermore, for the external distillation of features at different scales, we employ multi-scale convolutional attention to capture the relationships among features of different scales and ensure their consistency across old and new tasks. We evaluate our method on standard semantic segmentation datasets, such as Pascal-VOC2012 and ADE20K, and demonstrate significant performance improvements in various scenarios.
Location Attention Knowledge Embedding Model for Image-Text Matching

Image-text matching is the core algorithm of cross-modal retrieval, which plays a central role in connecting vision and text. Due to the well-known semantic gap between visual and textual modalities, yet image-text matching is a vital challenging task. In order to reduce the huge semantic difference between images and texts, existing methods use the consensus knowledge for image-text matching tasks. However, the consensus knowledge is only extracted based on the co-occurrence frequency of words in sentences in the corpus, and does not consider the semantic information contained in the image, resulting in a decline in semantic matching performance. To solve this issue, we propose a Location Attention Knowledge Embedding (LAKE) model to improve the consensus knowledge utilization by inferring the location of objects in an image. Specifically, our model consists of three parts: Firstly, we design a location feature extraction (LFE) module, which divides the image into blocks, uses the location attention to generate valuable location features, and then splices the location features with the extracted regional image features to obtain the image features containing location information. At the same time, text features are extracted using the BERT model. Secondly, we use a knowledge representation module to extract the consensus knowledge features. Finally, the similarity between the image and the text is calculated based on the knowledge fusion feature to complete the matching process. Quantitative and qualitative results on public datasets Flickr30k and MSCOCO demonstrate the effectiveness of the method.
Generalizable stereo depth estimation with masked image modelling

Generalizable and accurate stereo depth estimation is vital for 3D reconstruction, especially in surgery. Supervised learning methods obtain best performance however, limited ground truth data for surgical scenes limits generalizability. Self-supervised methods don't need ground truth, but suffer from scale ambiguity and incorrect disparity prediction due to inconsistency of photometric loss. This work proposes a two-phase training procedure that is generalizable and retains the high performance of supervised methods. It entails: (1) performing self-supervised representation learning of left and right views via masked image modelling (MIM) to learn generalizable semantic stereo features (2) utilizing the MIM pre-trained model to learn robust depth representation via supervised learning for disparity estimation on synthetic data only. To improve stereo representations learnt via MIM, perceptual loss terms are introduced, which improve the model's stereo representations learnt by explicitly encouraging the learning of higher scene-level features. Qualitative and quantitative performance evaluation on surgical and natural scenes shows that the approach achieves sub-millimetre accuracy and lowest errors respectively, setting a new state-of-the-art. Despite not training on surgical nor natural scene data for disparity estimation.