Reviewerly/exHarmony huggingface dataset:
UFO: Unified Fact Obtaining for Commonsense Question Answering

Leveraging external knowledge to enhance the reasoning ability is crucial for commonsense question answering. However, the existing knowledge bases heavily rely on manual annotation which unavoidably causes deficiency in coverage of world-wide commonsense knowledge. Accordingly, the knowledge bases fail to be flexible enough to support the reasoning over diverse questions. Recently, large-scale language models (LLMs) have dramatically improved the intelligence in capturing and leveraging knowledge, which opens up a new way to address the issue of eliciting knowledge from language models. We propose a Unified Facts Obtaining (UFO) approach. UFO turns LLMs into knowledge sources and produces relevant facts (knowledge statements) for the given question. We first develop a unified prompt consisting of demonstrations that cover different aspects of commonsense and different question styles. On this basis, we instruct the LLMs to generate question-related supporting facts for various commonsense questions via prompting. After facts generation, we apply a dense retrieval-based fact selection strategy to choose the best-matched fact. This kind of facts will be fed into the answer inference model along with the question. Notably, due to the design of unified prompts, UFO can support reasoning in various commonsense aspects (including general commonsense, scientific commonsense, and social commonsense). Extensive experiments on CommonsenseQA 2.0, OpenBookQA, QASC, and Social IQA benchmarks show that UFO significantly improves the performance of the inference model and outperforms manually constructed knowledge sources.
SwatchNet: Small Components Aware Attention for Fashion Product Recoloring

Automatic object recoloring or swatch generation aims to change color of an object or the entire scene without altering the structural consistency. The challenges exacerbate while dealing with retail items, such as clothing, shoes and accessories due to the presence of complex patterns, folds and shadows, deformation caused by the human model, and small components such as frills, buttons, belts, etc. The challenge associated with this application increases further for multi-colored item and multi-apparel setup. In this work, we aim to address these problems with our novel architecture SwatchNet based on generative adversarial network (GAN). It stems from the proposed apparel components aware feature extraction module to create rich feature embedding, which guides the proposed dual attention u-net to synthesize recolored product image. For seamless information flow, we have also proposed a dual attention module at the bottleneck of encoder and decoder. Finally, we unify a diverse set of recoloring applications using a fixed training and inference pipeline. The experimental results of fashion item recoloring for several test setups using four large-scale datasets demonstrate the effectiveness of our proposed approach.
Chinese Event Temporal Relation Extraction on Multi-Dimensional Attention

Extracting event temporal relations is an important task of natural language processing. It is even more challenging to extract the temporal relations of events without using annotated auxiliary information, which is time-consuming and expensive. Therefore, we propose an event temporal relation extraction model ETEMA for Chinese text based on BERT and multi-dimensional attention mechanism, using contextual information interaction and tensor matching methods. Specifically, ETEMA uses BERT to mine the semantic information of event sentences, and uses attention and semantic information to interactively combine event information with its contextual information. Experimental results on ACE2005-extended, a corpus of Chinese temporal relations, show that the proposed model ETEMA achieves optimal performance without using any auxiliary annotated information.
A Person Re-Identification Method Incorporating Multigrain Features and Human Body Knowledge

This paper generates a novel person re-identification (ReID) model that is trained with a joint loss function in a multi-branch structure, enabling the model to not only extract discriminative features but also focus on the extraction of foreground information, thereby improving its performance in the presence of occlusion. Numerous research has demonstrated that the multi-granularity features learned by the model can contain a wealth of information and details to characterize the entire image of a person, enabling the model to learn additional discriminative features and enhance its performance. An understanding of the human body can help the model extract human-specific features, which can reduce the interference of background noise and overcome the influence of body position variations to achieve the alignment of feature matching. In light of this, we present the ReID model MGHK (Mutil Granularity and Human Knowledge), which is comprised of global branches, fine-grained branches, and human key point branches. Through comparison experiments with a variety of representative ReID models, the results show that MGHK achieves better performance on several publicly available datasets. On Markrt1501, DukeMTMC, CUHK03 and MSMT17, Rankl reached 95.72%, 90.31%, 75.79% and 82.42%, respectively. mAP reached 87.09%, 77.53%, 72.95% and 59.31%, respectively. The Rankl reached 58.0% and 71.40% in the Partial-Reid and Occluded ReID datasets, respectively. mAP reached 57.68% and 66.21%. In addition, through Grad-CAM visualization analysis, extending MGHK's knowledge of human key points allows the acquisition of a broader diversity of human characteristics.
Partially-Labeled Domain Generalization via Multi-Dimensional Domain Adaptation

Domain generalization deals with a challenging setting where several labeled source domains are given, and the goal is to train machine learning models that can generalize to an unseen test domain. However, in practice, labeled samples are often difficult and expensive to obtain. Thus the source domains would not always be labeled. When only some source domains are labeled and others are unlabeled, we formally introduce this domain generalization problem as Partially-Labeled Domain Generalization (PLDG). In this paper, we study the most chal- lenging setting in PLDG problems, where only one source domain is labeled and a few unlabeled source domains are available. To enable generalization, we assume that all source domains follow certain domain index information that can reflect their domain relationships. With this domain index information, we propose a Multi-Dimensional Domain Adaptation (MDDA) method to address this PLDG problem. Specifically, the MDDA method first trains multiple domain adaptation models to adapt from the labeled source domain to all the unlabeled source domains via adversarial learning. Then those domain adaptation models and the source-only model trained on the labeled source domain only are distilled into the target model used for the unseen target domain. Theoretically, we provide a generalization bound of the MDDA method. The experiments on four real-world datasets demonstrate the effectiveness of the proposed MDDA method.
A Multi-Modal Approach for the Detection of Account Anonymity on Social Media Platforms

Anonymous users on Twitter are highly related to network hazards such as cyberbullying, fraud, rumor spread, etc. Given the inefficient and expensive manual review approach, an automated method to predict account anonymity is urgently needed. However, there are three challenges when employing deep learning algorithms to detect account anonymity: the lack of open-source public datasets, tedious manual collection and labeling for training data, and abundant types of information that the existing single-modal methods are not competent for. Our approach includes an automated labeling system for data collection and a multi-modal method for account anonymity detection. The labeling system is based on name extraction and identity verification and, to ensure the quality of the dataset, web mining and face similarity measurement are applied. We establish a dataset containing 20133 accounts and make it available to the public. The multi-modal method exploits the visual, textual, and numerical features extracted with ViT, BERT, and MLP, and uses the Transformer and MLP for feature fusion. Experimental results prove the effectiveness of the feature fusion and the accuracy of classification with accuracy of 86.21% and 92.46%.
One-step Domain Adaptation Approach with Partial Label

Unsupervised Domain adaptation (UDA) aims to train a target classifier by using massive accurate annotated source domain data and unlabeled target domain data. However, collecting massive accurate annotation can be labor-intensive, and even impractical especially when source domain training data shows label ambiguity. In this paper, we consider a novel domain adaptation setting where the model can be trained with partial-labeled source domain data, so that the cost of data labeling can be reduced. To alleviate the ambiguity induced by partial label, we propose a one-step domain adaptation approach trained from the partial-labeled source data and unlabeled target data. Our approach consists of two components: a feature extractor equipped with partial label loss that learns discriminative representation, and a domain classifier that learns domain-invariant representation across the source and target domains. Extensive experiments have shown that the proposed approach significantly outperforms a series of competitive baselines.
Improve Robustness of Graph Neural Networks: Multi-hop Neighbors Meet Homophily-based Truncation Defense

Graph Neural Networks (GNNs), as promising deep learning approaches, have been applied in various areas. However, it is also known that they are vulnerable to adversarial attacks, which raises many concerns in real application. Regarding Graph Structure Attacks (GSAs), though existing homophily-based truncation defenses have showed strong defense capacity, they suffer the problem of losing much effective neighborhood information in the process of removing adversarial edges, causing a limited performance on both clean and attacked graphs. In this paper, we consider the question: Can we capture more effective neighborhood information by utilizing the higher-order network to help improve the performance of the homophily-based truncation defense? To answer it, we first explore the impacts of different GSAs on the 1-hop and 2-hop networks. We theoretically and empirically find that the 2-hop network also has a strong information retention capacity like the 1-hop network after many GSAs. Motivated by this, we combine the 2-hop network with the homophily-based truncation defense, constructing a stronger defender, MHR-GCN. It integrates effective neighborhood information from both 1-hop and 2-hop networks. Extensive experiments demonstrate that MHR-GCN significantly improves the performance of the truncation defense and outperforms the state-of-the-arts under various GSA settings, especially when the graph is heavily perturbed.
Multi-dimensional Semantic-based Text Classification Model

In recent years, many excellent text classification models have been proposed by researchers to solve natural language processing tasks. In texts, semantic information at different levels and granularities constitutes the multi-dimensional semantic information of the text, which determines its classification effect. However, existing text classification models have several limitations. Firstly, they cannot extract the multi-dimensional semantic information from the text. Secondly, they cannot determine the weight of the multi-dimensional semantic information on the text's semantic information. To address these limitations, this paper proposes a new deep learning neural network model called the multi-dimensional semantic-based text classification model (MSTCM), which comprises an embedding module, a semantic processing module, and a prediction module. After processing the text through the embedding module, it passes through several sub-modules and layers in the semantic processing module to extract the multi-dimensional semantic information of the text and its weight on the text's semantic information using the attention mechanism. This information is used to generate a text vector, which is then utilized by the prediction module for text classification. To verify the effectiveness of MSTCM, experiments were conducted on several text classification tasks and compared with various text classification methods. The experimental results demonstrate that MSTCM performs better than other methods on most tasks.
“Find the Table”: A Contrastive Learning-based Approach with Faster RCNN for Establishing Tabular Entity Relationships

Financial industries rely on a variety of data to understand an organization's financial health and performance, and analysis of financial statements is used for decision-making and understanding business activities. Financial statements often have complex, unstructured formats, making it difficult to extract useful information for decision-making. Organizing and refining these data is crucial for effective analysis. Previous research in the field of table detection has primarily centred around object detection methods, with limited exploration of methods for cell-wise information extraction by identifying row and column entities. In order to address this research gap, this paper proposes a novel dataset called TERED (Tabular Entity Relationship Establishment Dataset) to train a model to identify relationships among elements in large financial tables, such as statements and balance sheets, using computer vision techniques which have yet to be fully explored in financial analysis. The dataset contains more than 10,000 tabular data in scanned image and pdf formats and is divided into 12 classes. We trained CLF-RCNN, a Contrastive Learning based Faster RCNN model which in turn is a state-of-the-art object detection model on this dataset and achieved an F1 score of 93 % for table detection and 74% for identifying tabular entities and relationships. Additionally, we introduced a new loss term influenced by contrastive learning that improves prediction performances with our developed algorithm to return the sequential order of the unordered predictions.
CAMVR: Context-Adaptive Multi-View Representation Learning for Dense Retrieval

The recently proposed MVR (Multi-View Repre-sentation) model achieves remarkable performance for open-domain dense retrieval. In MVR, matching multi-view queries is enabled by encoding the document into multiple representations. However, these representations tend to collapse into the same one when the percentage of documents answering multiple queries in the training data is low. Here, we propose a Context-Adaptive Multi-View Representation (CAMVR) learning framework that explicitly avoids this collapse problem by aligning each viewer token with different document snippets. In CAMVR, each viewer token is placed before a snippet to capture the local and global information with consideration that answers of different view queries may scatter in one document. The view of the snippet containing the answer is also used to explicitly supervise the learning process, from which the interpretability of the view representation is provided. Lastly, this work presents extensive experiments demonstrating that CAMVR outperforms the existing models to achieve state-of-the-art results.
A novel hardware-efficient liquid state machine of non-simultaneous CA-based neurons for spatio-temporal pattern recognition

In this paper, a novel liquid state machine (LSM) comprising neurons whose nonlinear dynamics are described by a non-simultaneous cellular automaton (CA) is proposed. The proposed LSM is applied to a supervised classification task. It is shown that the proposed LSM can recognize spatio-temporal spike patterns with high accuracy. Furthermore, the non-simultaneous CA-based neuron is implemented on a field programmable gate array (FPGA), and an experiment validates its spiking function. It is then shown that the non-simultaneous CA-based neuron occupies fewer FPGA resources compared with typical conventional neuron models, such as the Izhikevich, leaky integrate-and-fire, quadratic integrate-and-fire, and Morris–Lecar neurons.
Personalized Federated Learning with Multi-branch Architecture

Federated learning (FL) is a decentralized machine learning technique that enables multiple clients to collaboratively train models without requiring clients to reveal their raw data to each other. Although traditional FL trains a single global model with average performance among clients, statistical data heterogeneity across clients has resulted in the development of personalized FL (PFL), which trains personalized models with good performance on each client's data. A key challenge with PFL is how to facilitate clients with similar data to collaborate more in a situation where each client has data from complex distribution and cannot determine one another's distribution. In this paper, we propose a new PFL method (pFedMB) using multi-branch architecture, which achieves personalization by splitting each layer of a neural network into multiple branches and assigning client-specific weights to each branch. We also design an aggregation method to improve the communication efficiency and the model performance, with which each branch is globally updated with weighted averaging by client-specific weights assigned to the branch. pFedMB is simple but effective in facilitating each client to share knowledge with similar clients by adjusting the weights assigned to each branch. We experimentally show that pFedMB performs better than the state-of-the-art PFL methods using the CIFAR10 and CIFAR100 datasets.
SparGE: Sparse Coding-based Patient Similarity Learning via Low-rank Constraints and Graph Embedding

Patient similarity assessment (PSA) is pivotal to evidence-based and personalized medicine, enabled by analyzing the increasingly available electronic health records (EHRs). However, machine learning approaches for PSA have to deal with inherent data deficiencies of EHRs, namely missing values, noise, and small sample sizes. In this work, an end-to-end discriminative learning framework, called SparGE, is proposed to address these data challenges of EHR for PSA. SparGE measures similarity by jointly sparse coding and graph embedding. First, we use low-rank constrained sparse coding to identify and calculate weight for similar patients, while denoising against missing values. Then, graph embedding on sparse representations is adopted to measure the similarity between patient pairs via preserving local relationships defined by distances. Finally, a global cost function is constructed to optimize related parameters. Experimental results on two private and public real-world healthcare datasets, namely SingHEART and MIMIC-III, show that the proposed SparGE significantly outperforms other machine learning patient similarity methods.
Time-domain Fading Channel Prediction Based on Spin-wave Reservoir Computing

This paper proposes a physical-device-based time-domain fading channel prediction scheme using spin-wave reservoir computing. We numerically construct a spin-wave reservoir chip that adopts new spin-wave transducers named Film-penetrating transducers (FPTs). We arrange three FPTs as exciters and forty-nine FPTs as detectors connected to the reservoir input and readout, respectively. We feed communication channel information collected in an actual fading environment to evaluate the prediction performance. We calculate the average root mean squared errors (RMSEs) and obtain the symbol error rates (SERs) for various forecasting time lengths. We find that our proposed scheme can achieve accurate channel prediction without any frequency-domain conversion. We obtain robust communication performance up to a forecasting time length of 8 ms. These results suggest a high capability of spin-wave reservoir computing in the application of channel prediction and its promising potential in other possible time-sequential computational tasks.
Grid Feature Jigsaw for Self-supervised Image Clustering

Image clustering is an essential unsupervised learning task in computer vision. The key issue for image clustering is to learn the representative visual features without annotations to extend class spacing. Jigsaw puzzles, as one pretext task of self-supervised visual representation learning to learn the relative spatial position of image tiles, has attracted the attention of many researchers. Most of existing jigsaw puzzle solving strategies are based on the raw image patches, i.e. original pixels, which makes them only concentrate on the low-level statistics. In this paper, we propose a novel learning strategy named Grid Feature Jigsaw (GFJ) for self-supervised image clustering to increase class spacing by deep mining of single sample feature. We train the model to learn the intra-grid representation via the self-supervised paradigm. By dividing the feature map into grids and arranging adjacent grids in each block, we implement a linear regression from the surrounding grids to represent the reference grid. The experiments of unsupervised computer vision benchmark show the effectiveness on the clustering task with respect to the ACC, NMI and ARI three metrics and we verify GFJ universal performance via various deep convolutional neural networks.
Insty-GAN for Image-to-Image Location Transfer

Image-to-Image Transformation (I2I) aims to transfer an image from a source domain to a target domain while changing some specified feature representations and preserving some other feature representations. For example, most I2I tasks keep the layout features of an image while changing its color, style, or resolution features. However, image-to-image location transfer only wants to change the layout of objects in an image's foreground, such as LT-GAN and Orthod-GAN. In this paper, we researched the above image-to-image location translation tasks. First, we studied the similarities and differences between malpositioned-teeth images and their corresponding neat-teeth images. Then we analyzed the problems of directly applying the LT-GAN method to the orthodontics transfer task. Finally, we proposed a new approach called Insty-GAN that uses an instance transfer bridge and a style loss to address image-to-image location transfer. Our Insty-GAN is applied to orthodontic and chaos-to-order transfer tasks and improved the FID score by 9.1% and 8.1% over their SOTA methods. LPIPS scores also show that Insty-GAN performs a more excellent production than other baselines in image-to-image location transfer. Additionally, we have annotated 10K instance segmentation labels for Orthod and M2C datasets. These labels could also be used in future works. The added data is available at https://drive.google.com/drive/folders/l5ilnLE_pwIB6fQq1nbWTpMxav6COdU7p?usp=sharing.
Towards Efficient Knowledge Reuse for Open-ended Learning in Real Robots through Motivation

The current work is focused on providing robotic systems with mechanisms that support lifelong open-ended learning, with the aim of increasing their real autonomy level. In this general scope, robots must discover their goals and learn the skills to achieve them in a priori unknown domains and tasks. Moreover, they must do it in way that allows this knowledge to be reused later to face new situations properly. To advance in this challenging field, this paper describes a specific motivation-based knowledge reuse strategy, together with the contextual processing carried out, within the e-MDB cognitive architecture. This strategy supports the reuse and adaptation of knowledge acquired in previously seen domains to new ones. It has been validated in a real-world experiment with the Baxter robot, which is analyzed and discussed here, that addresses open-ended interaction in a sequence of domains related to object manipulation.