Reviewerly/exHarmony huggingface dataset:
Semantic Line Detection Using Deep-Hough with Attention Mechanism and Strip Convolution

Semantic lines are some particular dominant lines in an image, which divide the image into several semantic regions and outline its conceptual structure. They play a vital role in image analysis, scene understanding, and other downstream tasks due to their semantic representation ability for the image layout. However, the accuracy and efficiency of existing semantic line detection methods still couldn't meet the need of real applications. So, a new semantic line detection method based on the deep Hough transform network with attention mechanism and strip convolution is proposed. Firstly, the detection performance is improved by combining the channel attention mechanism with the feature pyramid network to alleviate the influence of redundant information. Then, the strip convolution and mixed pooling layer are introduced to effectively collect the remote information and capture the long-range dependencies between pixel backgrounds. Finally, the strategy of GhostNet is adopted to reduce the computational cost. Results of experiments on open datasets validate the proposed method, which is comparable to and even outperforms the state-of-the-art methods in accuracy and efficiency. Our code and pretrained models are available at: https://github.com/zhizhz/sml.
YOLOMM – You Only Look Once for Multi-modal Multi-tasking

Autonomous driving can reduce the number of road accidents due to human error and result in safer roads. One important part of the system is the perception unit, which provides information about the environment surrounding the car. Currently, most manufacturers are using not only RGB cameras, which are passive sensors that capture light already in the environment but also Lidar. This sensor actively emits laser pulses to a surface or object and measures reflection and time-of-flight. Previous work, YOLOP, already proposed a model for object detection and semantic segmentation, but only using RGB. This work extends it for Lidar and evaluates performance on KITTI, a public autonomous driving dataset. The implementation shows improved precision across all objects of different sizes. The implementation is entirely made available: https://github.com/filipepcampos/yolomm .
How to Turn Your Camera into a Perfect Pinhole Model

Camera calibration is a first and fundamental step in various computer vision applications. Despite being an active field of research, Zhang’s method remains widely used for camera calibration due to its implementation in popular toolboxes like MATLAB and OpenCV. However, this method initially assumes a pinhole model with oversimplified distortion models. In this work, we propose a novel approach that involves a pre-processing step to remove distortions from images by means of Gaussian processes. Our method does not need to assume any distortion model and can be applied to severely warped images, even in the case of multiple distortion sources, e.g., a fisheye image of a curved mirror reflection. The Gaussian processes capture all distortions and camera imperfections, resulting in virtual images as though taken by an ideal pinhole camera with square pixels. Furthermore, this ideal GP-camera only needs one image of a square grid calibration pattern. This model allows for a serious upgrade of many algorithms and applications that are designed in a pure projective geometry setting but with a performance that is very sensitive to non-linear lens distortions. We demonstrate the effectiveness of our method by simplifying Zhang’s calibration method, reducing the number of parameters and getting rid of the distortion parameters and iterative optimization. We validate by means of synthetic data and real world images. The contributions of this work include the construction of a virtual ideal pinhole camera using Gaussian processes, a simplified calibration method and lens distortion removal.
Diachronic Named Entity Disambiguation for Ancient Chinese Historical Records

Named entity disambiguation (NED) is a fundamental task in NLP. Although numerous methods have been proposed for NED in recent years, they ignore the fact that a lot of real-world corpora are diachronic by nature, such as historical documents or news articles, which vary greatly in time. As a consequence, most current methods fail to fully exploit the temporal information inside the corpora and knowledge bases. To address the issue, we propose a novel model which integrates temporal feature into pretrained language model to make our model aware of time and a new sample re-weighting scheme for diachronic NED which penalizes highly-frequent mention-entity pairs to improve performance on rare and unseen entities. We present WikiCMAG and WikiSM, two new NED datasets annotated on ancient Chinese historical records. Experiments show that our model outperforms existing methods by large margins, proving the effectiveness of integrating diachronic information and our re-weighting schema. Our model also gains competitive performance on out-of-distribution (OOD) settings. WikiSM is publicly available at https://github.com/PKUDHC/WikiSM .
ASRCD: Adaptive Serial Relation-Based Model for Cognitive Diagnosis

Cognitive diagnosis (CD) is a critical task in the education field, aimed at assessing the true concept proficiency of learners. Recent studies have highlighted the significance of concept relations (e.g., concept Addition and concept Multiplication in mathematics) in CD. While advanced research has contributed to concept relation modeling, there remains a gap in automatic building and adaptive integration of relation modeling. To address these challenges, we present an innovative approach called the Adaptive Serial Relation-based model for Cognitive Diagnosis (ASRCD). Our method begins by constructing a Concept Serial Relation Graph (CSRG) to automatically mine concept relations from learner response sequences. Next, a refined graph attention network (GAT) is designed to weight the concept relations for effective aggregation. Finally, we establish a comprehensive CD model that incorporates concept relations, leveraging the extendibility of CSRG, allowing it to be seamlessly integrated into various existing CD methods. To evaluate the performance of our model, we conduct experiments on two real-world datasets from education practice. The experimental results demonstrate that our proposed ASRCD model achieves outstanding accuracy and exhibits excellent extendibility, showcasing its effectiveness and potential in enhancing cognitive diagnosis tasks.
PyraBiNet: A Hybrid Semantic Segmentation Network Combining PVT and BiSeNet for Deformable Objects in Indoor Environments

In this study, we introduce PyraBiNet, an innovative hybrid model optimized for lightweight semantic segmentation tasks. This model ingeniously merges the merits of Convolutional Neural Networks (CNNs) and Transformers. We propose a dual-branch structure that strategically employs the global feature extraction capabilities of the Pyramidal Vision Transformer (PVT) and the local feature extraction proficiency of BiSeNet. Specifically, the global feature branch employs a transformer from PVT to harness high-level patterns from input images, while the local feature branch utilizes a CNN, inspired by BiSeNet, to extract fine-grained details. Comprehensive evaluations conducted on the ADE20K and DOS datasets underscore PyraBiNet's superior performance compared to the existing state-of-the-art methods. With its effective and efficient performance, PyraBiNet proves to be an invaluable asset in the domain of mobile robotics, particularly beneficial for applications such as sweeping robots. The code source and dataset are open at https://github.com/zehantan6970/PyraBiNet .
Presumably Correct Undersampling

This paper presents a data pre-processing algorithm to tackle class imbalance in classification problems by undersampling the majority class. It relies on a formalism termed Presumably Correct Decision Sets aimed at isolating easy (presumably correct) and difficult (presumably incorrect) instances in a classification problem. The former are instances with neighbors that largely share their class label, while the latter have neighbors that mostly belong to a different decision class. The proposed algorithm replaces the presumably correct instances belonging to the majority decision class with prototypes, and it operates under the assumption that removing these instances does not change the boundaries of the decision space. Note that this strategy opposes other methods that remove pairs of instances from different classes that are each other’s closest neighbors. We argue that the training and test data should have similar distribution and complexity and that making the decision classes more separable in the training data would only increase the risks of overfitting. The experiments show that our method improves the generalization capabilities of a baseline classifier, while outperforming other undersampling algorithms reported in the literature.
Graph Attention Hashing via Contrastive Learning for Unsupervised Cross-Modal Retrieval

Hashing-based cross-modal retrieval maps multi-modal features into binary codes into a common Hamming space. Due to its small storage consumption and high efficiency, hashing has received extensive attention in recent years. However, the current researches have difficulty in constructing a well-defined joint semantic space and conduct more detailed and in-depth learning guidance. In this paper, Graph Attention Hashing via Contrastive Learning (GAHCL) is proposed to address these issues. First, we use the idea of contrastive learning to generate positive samples, and propose a novel contrastive adjacency matrix through a graph attention network. Specifically, this matrix assigns higher weights to node pairs whose source is the same sample, and assigns lower weights to node pairs that do not match each other. The key semantic features can be captured more carefully and accurately under the influence of attention weights. In addition, the contrastive loss function is constructed by taking the output features of different modalities in an instance and its generated positive sample features as a positive sample pair. Extensive experiments on two datasets show that the proposed method can significantly outperform existing competitors.
IR-Guided Energy Optimization Framework for Depth Enhancement in Time of Flight Imaging

This paper introduces an optimization energy framework based on infrared guidance to improve depth consistency in Time of Flight image systems. The primary objective is to formulate the problem as an image energy optimization task, aimed at maximizing the coherence between the depth map and the corresponding infrared image, both captured simultaneously from the same Time of Flight sensor. The concept of depth consistency relies on the underlying hypothesis concerning the correlation between depth maps and their corresponding infrared images. The proposed optimization framework adopts a weighted approach, leveraging an iterative estimator. The image energy is characterized by introducing spatial conditional entropy as a correlation measure and spatial error as image regularization. To address the issue of missing depth values, a preprocessing step is initially applied, by using a depth completion method based on infrared guided belief propagation, which was proposed in a previous work. Subsequently, the proposed framework is employed to regularize and enhance the inpainted depth. The experimental results demonstrate a range of qualitative improvements in depth map reconstruction, with a particular emphasis on the sharpness and continuity of edges.
Multi-Task Feature Self-Distillation for Semi-Supervised Machine Translation

The performance of the model suffers dramatically when the scale of supervision data is constrained because the most sophisticated neural machine translation models are all data-driven techniques. Adding monolingual data to the model training process and using the back-translation method to create pseudo-parallel sentences for data augmentation is the current accepted approach to address data shortage. However, this method's training procedures are laborious, it takes a lot of time, and some of the created pseudo-parallel sentences are of poor quality and detrimental to the model. In this paper, we propose a semi-supervised training method—Multi-Task Feature Self-Distillation (MFSD), which can train models online on a mixed dataset consisting of bilingual and monolingual data jointly. Based on the supervised machine translation task, we propose a self-distillation task for the encoder and decoder of the source and target languages to train two kinds of monolingual data online. In the self-distillation task, we build a teacher model by integrating the student models of the previous rounds, and then use the feature soft labels output by the teacher model with more stable performance to guide the student model training online, and realize the online mining single High-level knowledge of language data by comparing the consistency of the output features of the two models. We conduct experiments on the standard data set and the multi-domain translation data set respectively. The results show that MFSD performed better than mainstream semi-supervised and data augmentation methods approaches. Compared with supervised training, our method improves the BLEU score by an average of +2.27, and effectively improves the model's domain adaptability and generalization ability.
A Two-Stage Active Learning Algorithm for NLP Based on Feature Mixing

Active learning (AL) aims to improve the model performance with minimal data annotation. While recent AL studies have utilized feature mixing to identify unlabeled instances with novel features, applying it to natural language processing (NLP) tasks has been challenged due to the discrete nature of text tokens and the limited contribution of some novel features. To address these issues, we propose a two-stage acquisition method based on feature mixing for NLP tasks. We first create a mixed feature for both labeled and unlabeled instances to identify the features in the unlabeled instances that the model cannot recognize. Next, we evaluate the contribution of these novel features to the model using the entropy of the nearest labeled neighbors. The proposed method enables the model to select the most informative samples in the unlabeled sample pool. Experiments on sentiment analysis, topic classification, and natural language inference validated that our method not only outperforms other AL approaches but improves the efficiency of batch data acquisition.
Continual Domain Adaption for Neural Machine Translation

Domain Neural Machine Translation (NMT) with small data- sets requires continual learning to incorporate new knowledge, as catastrophic forgetting is the main challenge that causes the model to forget old knowledge during fine-tuning. Additionally, most studies ignore the multi-stage domain adaptation of NMT. To address these issues, we propose a multi-stage incremental framework for domain NMT based on knowledge distillation. We also analyze how the supervised signals of the golden label and the teacher model work within a stage. Results show that the teacher model can only benefit the student model in the early epochs, while harms it in the later epochs. To solve this problem, we propose using two training objectives to encourage the early and later training. For early epochs, conventional continual learning is retained to fully leverage the teacher model and integrate old knowledge. For the later epochs, the bidirectional marginal loss is used to get rid of the negative impact of the teacher model. The experiments show that our method outperforms multiple continual learning methods, with an average improvement of 1.11 and 1.06 on two domain translation tasks.
Category-Wise Fine-Tuning for Image Multi-label Classification with Partial Labels

Image multi-label classification datasets are often partially labeled (for each sample, only the labels on some categories are known). One popular solution for training convolutional neural networks is treating all unknown labels as negative labels, named Negative mode. But it produces wrong labels unevenly over categories, decreasing the binary classification performance on different categories to varying degrees. On the other hand, although Ignore mode that ignores the contributions of unknown labels may be less effective than Negative mode, it ensures the data have no additional wrong labels, which is what Negative mode lacks. In this paper, we propose Category-wise Fine-Tuning (CFT), a new post-training method that can be applied to a model trained with Negative mode to improve its performance on each category independently. Specifically, CFT uses Ignore mode to one-by-one fine-tune the logistic regressions (LRs) in the classification layer. The use of Ignore mode reduces the performance decreases caused by the wrong labels of Negative mode during training. Particularly, Genetic Algorithm (GA) and binary crossentropy are used in CFT for fine-tuning the LRs. The effectiveness of our methods was evaluated on the CheXpert competition dataset and achieves state-of-the-art results, to our knowledge. A single model submitted to the competition server for the official evaluation achieves mAUC 91.82% on the test set, which is the highest single model score in the leaderboard and literature. Moreover, our ensemble achieves mAUC 93.33% (The competition was recently closed. We evaluate the ensemble on a local machine after the test set is released and can be downloaded.) on the test set, superior to the best in the leaderboard and literature (93.05%). Besides, the effectiveness of our methods is also evaluated on the partially labeled versions of the MS-COCO dataset.
Neural-Symbolic Reasoning with External Knowledge for Machine Reading Comprehension

Machine reading comprehension is a fundamental in natural language understanding. Existing large-scale pre-trained language models and graph neural network-based models have achieved good gains on logical reasoning of text. However, neither of them can give a complete reasoning chain, while symbolic logic-based reasoning is explicit and explainable. Therefore, we propose a framework LoGEK that integrates symbolic Logic and Graph neural networks for reasoning, while leveraging External Knowledge to augment the logical graph. The LoGEK model consists of three parts: logic extraction and extension, logical graph reasoning and answer prediction. Specifically, LoGEK extracts and extends logic set from the unstructured text. Then the logical graph reasoning module uses external knowledge to extend the original logical graph. After that, the model uses a path-based relational graph neural network to model the extended logical graph. Finally, the prediction module performs answer prediction based on graph embeddings and text embeddings. We conduct experiments on benchmark datasets for logical reasoning to evaluate the performance of LoGEK. The experimental results show that the accuracy of the method in this paper is better than the baseline models, which verifies the effectiveness of the method.
A Self-Organizing Map Clustering Approach to Support Territorial Zoning

This work aims to evaluate three strategies for analyzing clusters of ordinal categorical data (thematic maps) to support the territorial zoning of the Alto Taquari basin, MS/MT. We evaluated a model-based method, another based on the segmentation of the multi-way contingency table, and the last one based on the transformation of ordinal data into intervals and subsequent analysis of clusters from a proposed method of segmentation of the Self-Organizing Map after the neural network training process. The results showed the adequacy of the methods based on the Self-Organizen Map and the segmentation of the contingency table, as these techniques generated unimodal clusters with distinguishable groups.
Unveiling the Influence of Image Super-Resolution on Aerial Scene Classification

Deep learning has made significant advances in recent years, and as a result, it is now in a stage where it can achieve outstanding results in tasks requiring visual understanding of scenes. However, its performance tends to decline when dealing with low-quality images. The advent of super-resolution (SR) techniques has started to have an impact on the field of remote sensing by enabling the restoration of fine details and enhancing image quality, which could help to increase performance in other vision tasks. However, in previous works, contradictory results for scene visual understanding were achieved when SR techniques were applied. In this paper, we present an experimental study on the impact of SR on enhancing aerial scene classification. Through the analysis of different state-of-the-art SR algorithms, including traditional methods and deep learning-based approaches, we unveil the transformative potential of SR in overcoming the limitations of low-resolution (LR) aerial imagery. By enhancing spatial resolution, more fine details are captured, opening the door for an improvement in scene understanding. We also discuss the effect of different image scales on the quality of SR and its effect on aerial scene classification. Our experimental work demonstrates the significant impact of SR on enhancing aerial scene classification compared to LR images, opening new avenues for improved remote sensing applications.
Light Field Image Super-Resolution via Global-View Information Adaption and Angular Attention Fusion

Light field (LF) is a emerging technology, which can be used in many fields. Furthermore, LF cameras can capture spatial and angular information of 3D real-world scenes. This information is beneficial for image super-resolution (SR). However, most existing LF approaches have the limitation of utilizing the global-view information, which contains the correlation information among all LF. Moreover, to exploit the complementary information from different views of an LF image, we propose a novel SR method that adapts each view to a global domain with the guidance of global-view information. Our method, called LF-GAGNet, uses a dual-branch network to align features across views with deformable convolutions and fuse them with an attention mechanism. The upper branch extracts global-view information and generates adaptive guidance factors for each view through a global-view adaptation-guided module (GAGM). The lower branch uses these factors as offsets for deformable convolutions to achieve feature alignment in the global domain. Furthermore, we design an angular attention fusion module (AAFM) to enhance the angular features of each view according to their importance. We evaluate our method on various real-world scenarios and show that it surpasses other state-of-the-art methods in terms of SR quality and performance. We also demonstrate that our method can handle complex realistic LF scenarios effectively.
Leveraging Question Answering for Domain-Agnostic Information Extraction

Transformers gave a considerable boost to Natural Language Processing, but their application to specific scenarios still poses some practical issues. We present an approach for extracting information from technical documents on different domains, with minimal effort. It leverages on generic models for Question Answering and on questions formulated with target properties in mind. These are made to specific sections where the answer, then used as the value for the property, should reside. We further describe how this approach was applied to documents of two very different domains: toxicology and finance. For both, results extracted from a sample of documents were assessed by domain experts, who also provided feedback on the benefits of this approach. F-Scores of 0.73 and 0.90, respectively in the toxicological and financial domain, confirm the potential and flexibility of the approach suggesting that, while it cannot yet be fully automated and replace human work, it can support expert decisions, thus reducing time and manual effort.