OpenRaiser/Intern-Atlas huggingface dataset:
Multigranular Visual-Semantic Embedding for Cloth-Changing Person Re-identification
Venue: arXiv cs.CV 2021
Year: 2021

Person reidentification (ReID) is a very hot research topic in machine learning and computer vision, and many person ReID approaches have been proposed; however, most of these methods assume that the same person has the same clothes within a short time interval, and thus their visual appearance must be similar. However, in an actual surveillance environment, a given person has a great probability of changing clothes after a long time span, and they also often take different personal belongings with them. When the existing person ReID methods are applied in this type of case, almost all of them fail. To date, only a few works have focused on the cloth-changing person ReID task, but since it is very difficult to extract generalized and robust features for representing people with different clothes, their performances need to be improved. Moreover, visual-semantic information is often ignored. To solve these issues, in this work, a novel multigranular visual-semantic embedding algorithm (MVSE) is proposed for cloth-changing person ReID, where visual semantic information and human attributes are embedded into the network, and the generalized features of human appearance can be well learned to effectively solve the problem of clothing changes. Specifically, to fully represent a person with clothing changes, a multigranular feature representation scheme (MGR) is employed to focus on the unchanged part of the human, and then a cloth desensitization network (CDN) is designed to improve the feature robustness of the approach for the person with different clothing, where different high-level human attributes are fully utilized. Moreover, to further solve the issue of pose changes and occlusion under different camera perspectives, a partially semantically aligned network (PSA) is proposed to obtain the visual-semantic information that is used to align the human attributes.
Accelerating Evolutionary Neural Architecture Search via Multi-Fidelity Evaluation
Venue: arXiv cs.AI 2021
Year: 2021

Evolutionary neural architecture search (ENAS) has recently received increasing attention by effectively finding high-quality neural architectures, which however consumes high computational cost by training the architecture encoded by each individual for complete epochs in individual evaluation. Numerous ENAS approaches have been developed to reduce the evaluation cost, but it is often difficult for most of these approaches to achieve high evaluation accuracy. To address this issue, in this paper we propose an accelerated ENAS via multifidelity evaluation termed MFENAS, where the individual evaluation cost is significantly reduced by training the architecture encoded by each individual for only a small number of epochs. The balance between evaluation cost and evaluation accuracy is well maintained by suggesting a multi-fidelity evaluation, which identifies the potentially good individuals that cannot survive from previous generations by integrating multiple evaluations under different numbers of training epochs. For high diversity of neural architectures, a population initialization strategy is devised to produce different neural architectures varying from ResNet-like architectures to Inception-like ones. Experimental results on CIFAR-10 show that the architecture obtained by the proposed MFENAS achieves a 2.39% test error rate at the cost of only 0.6 GPU days on one NVIDIA 2080TI GPU, demonstrating the superiority of the proposed MFENAS over state-of-the-art NAS approaches in terms of both computational cost and architecture quality. The architecture obtained by the proposed MFENAS is then transferred to CIFAR-100 and ImageNet, which also exhibits competitive performance to the architectures obtained by existing NAS approaches. The source code of the proposed MFENAS is available at https://github.com/DevilYangS/MFENAS/.
Adaptive Beam Tracking based on Recurrent Neural Networks for mmWave Channels
Venue: arXiv cs.IT 2021
Year: 2021

The performance of millimeter wave (mmWave) communications critically depends on the accuracy of beamforming both at base station (BS) and user terminals (UEs) due to high isotropic path-loss and channel attenuation. In high mobility environments, accurate beam alignment becomes even more challenging as the angles of the BS and each UE must be tracked reliably and continuously. In this work, focusing on the beamforming at the BS, we propose an adaptive method based on Recurrent Neural Networks (RNN) that tracks and predicts the Angle of Departure (AoD) of a given UE. Moreover, we propose a modified frame structure to reduce beam alignment overhead and hence increase the communication rate. Our numerical experiments in a highly non-linear mobility scenario show that our proposed method is able to track the AoD accurately and achieve higher communication rate compared to more traditional methods such as the particle filter.
3D Visualisations Should Not be Displayed Alone - Encouraging a Need for Multivocality in Visualisation
Venue: arXiv cs.HC 2021
Year: 2021

We believe that 3D visualisations should not be used alone; by coincidentally displaying alternative views the user can gain the best understanding of all situations. The different presentations signify manifold meanings and afford different tasks. Natural 3D worlds implicitly tell many stories. For instance, walking into a living room, seeing the TV, types of magazines, pictures on the wall, tells us much about the occupiers: their occupation, standards of living, taste in design, whether they have kids, and so on. How can we similarly create rich and diverse 3D visualisation presentations? How can we create visualisations that allow people to understand different stories from the data? In a multivariate 2D visualisation a developer may coordinate and link many views together to provide exploratory visualisation functionality. But how can this be achieved in 3D and in immersive visualisations? Different visualisation types, each have specific uses, and each has the potential to tell or evoke a different story. Through several use-cases, we discuss challenges of 3D visualisation, and present our argument for concurrent and coordinated visualisations of alternative styles, and encourage developers to consider using alternative representations with any 3D view, even if that view is displayed in a virtual, augmented or mixed reality setup.
Tutorial on the Robust Interior Point Method
Venue: arXiv cs.DS 2021
Year: 2021

We give a short, self-contained proof of the interior point method and its robust version.
Semantics-STGCNN: A Semantics-guided Spatial-Temporal Graph Convolutional Network for Multi-class Trajectory Prediction
Venue: arXiv cs.CV 2021
Year: 2021

Predicting the movement trajectories of multiple classes of road users in real-world scenarios is a challenging task due to the diverse trajectory patterns. While recent works of pedestrian trajectory prediction successfully modelled the influence of surrounding neighbours based on the relative distances, they are ineffective on multi-class trajectory prediction. This is because they ignore the impact of the implicit correlations between different types of road users on the trajectory to be predicted - for example, a nearby pedestrian has a different level of influence from a nearby car. In this paper, we propose to introduce class information into a graph convolutional neural network to better predict the trajectory of an individual. We embed the class labels of the surrounding objects into the label adjacency matrix (LAM), which is combined with the velocity-based adjacency matrix (VAM) comprised of the objects' velocity, thereby generating a semantics-guided graph adjacency (SAM). SAM effectively models semantic information with trainable parameters to automatically learn the embedded label features that will contribute to the fixed velocity-based trajectory. Such information of spatial and temporal dependencies is passed to a graph convolutional and temporal convolutional network to estimate the predicted trajectory distributions. We further propose new metrics, known as Average2 Displacement Error (aADE) and Average Final Displacement Error (aFDE), that assess network accuracy more accurately. We call our framework Semantics-STGCNN. It consistently shows superior performance to the state-of-the-arts in existing and the newly proposed metrics.
Analysis of ODE2VAE with Examples
Venue: arXiv cs.LG 2021
Year: 2021

Deep generative models aim to learn underlying distributions that generate the observed data. Given the fact that the generative distribution may be complex and intractable, deep latent variable models use probabilistic frameworks to learn more expressive joint probability distributions over the data and their low-dimensional hidden variables. Learning complex probability distributions over sequential data without any supervision is a difficult task for deep generative models. Ordinary Differential Equation Variational Auto-Encoder (ODE2VAE) is a deep latent variable model that aims to learn complex distributions over high-dimensional sequential data and their low-dimensional representations. ODE2VAE infers continuous latent dynamics of the high-dimensional input in a low-dimensional hierarchical latent space. The hierarchical organization of the continuous latent space embeds a physics-guided inductive bias in the model. In this paper, we analyze the latent representations inferred by the ODE2VAE model over three different physical motion datasets: bouncing balls, projectile motion, and simple pendulum. Through our experiments, we explore the effects of the physics-guided inductive bias of the ODE2VAE model over the learned dynamical latent representations. We show that the model is able to learn meaningful latent representations to an extent without any supervision.
A mechanism-based gradient damage model for metallic fracture
Venue: arXiv cs.CE 2021
Year: 2021

A new gradient-based formulation for predicting fracture in elastic-plastic solids is presented. Damage is captured by means of a phase field model that considers both the elastic and plastic works as driving forces for fracture. Material deformation is characterised by a mechanism-based strain gradient constitutive model. This non-local plastic-damage formulation is numerically implemented and used to simulate fracture in several paradigmatic boundary value problems. The case studies aim at shedding light into the role of the plastic and fracture length scales. It is found that the role of plastic strain gradients is two-fold. When dealing with sharp defects like cracks, plastic strain gradients elevate local stresses and facilitate fracture. However, in the presence of non-sharp defects failure is driven by the localisation of plastic flow, which is delayed due to the additional work hardening introduced by plastic strain gradients.
BERTHop: An Effective Vision-and-Language Model for Chest X-ray Disease Diagnosis
Venue: arXiv cs.CV 2021
Year: 2021

Vision-and-language(V&L) models take image and text as input and learn to capture the associations between them. Prior studies show that pre-trained V&L models can significantly improve the model performance for downstream tasks such as Visual Question Answering (VQA). However, V&L models are less effective when applied in the medical domain (e.g., on X-ray images and clinical notes) due to the domain gap. In this paper, we investigate the challenges of applying pre-trained V&L models in medical applications. In particular, we identify that the visual representation in general V&L models is not suitable for processing medical data. To overcome this limitation, we propose BERTHop, a transformer-based model based on PixelHop++ and VisualBERT, for better capturing the associations between the two modalities. Experiments on the OpenI dataset, a commonly used thoracic disease diagnosis benchmark, show that BERTHop achieves an average Area Under the Curve (AUC) of 98.12% which is 1.62% higher than state-of-the-art (SOTA) while it is trained on a 9 times smaller dataset.
Knowledge-Based Stable Roommates Problem: A Real-World Application
Venue: arXiv cs.AI 2021
Year: 2021

The Stable Roommates problem with Ties and Incomplete lists (SRTI) is a matching problem characterized by the preferences of agents over other agents as roommates, where the preferences may have ties or be incomplete. SRTI asks for a matching that is stable and, sometimes, optimizes a domain-independent fairness criterion (e.g., Egalitarian). However, in real-world applications (e.g., assigning students as roommates at a dormitory), we usually consider a variety of domain-specific criteria depending on preferences over the habits and desires of the agents. With this motivation, we introduce a knowledge-based method to SRTI considering domain-specific knowledge, and investigate its real-world application for assigning students as roommates at a university dormitory. This paper is under consideration for acceptance in Theory and Practice of Logic Programming (TPLP).
Science Tree: A Platform for Exploring the Brazilian Academic Genealogy
Venue: arXiv cs.DL 2021
Year: 2021

Identifying and studying the formation of researchers over the years is a challenging task, as the current repositories of theses and dissertations are cataloged in a decentralized manner in different digital libraries, many of them with limited scope. In this paper, we take a step forward towards building a large repository to record the Brazilian academic genealogy. For this, we collected data from the Lattes platform, an internationally recognized initiative that provides a repository of researchers' curricula maintained by the Brazilian National Council for Scientific and Technological Development (CNPq), and developed a user-oriented platform to generate the academic genealogy trees of Brazilian researchers from them, also providing additional data resulting from a series of analyses regarding the main properties of such trees. Our effort has identified interesting aspects related to the academic career of the Brazilian researchers, which highlight the importance of generating and cataloging their academic genealogy trees.
Learning Fair Face Representation With Progressive Cross Transformer
Venue: arXiv cs.CV 2021
Year: 2021

Face recognition (FR) has made extraordinary progress owing to the advancement of deep convolutional neural networks. However, demographic bias among different racial cohorts still challenges the practical face recognition system. The race factor has been proven to be a dilemma for fair FR (FFR) as the subject-related specific attributes induce the classification bias whilst carrying some useful cues for FR. To mitigate racial bias and meantime preserve robust FR, we abstract face identity-related representation as a signal denoising problem and propose a progressive cross transformer (PCT) method for fair face recognition. Originating from the signal decomposition theory, we attempt to decouple face representation into i) identity-related components and ii) noisy/identity-unrelated components induced by race. As an extension of signal subspace decomposition, we formulate face decoupling as a generalized functional expression model to cross-predict face identity and race information. The face expression model is further concretized by designing dual cross-transformers to distill identity-related components and suppress racial noises. In order to refine face representation, we take a progressive face decoupling way to learn identity/race-specific transformations, so that identity-unrelated components induced by race could be better disentangled. We evaluate the proposed PCT on the public fair face recognition benchmarks (BFW, RFW) and verify that PCT is capable of mitigating bias in face recognition while achieving state-of-the-art FR performance. Besides, visualization results also show that the attention maps in PCT can well reveal the race-related/biased facial regions.
Cybersecurity Incident Response in Organisations: A Meta-level Framework for Scenario-based Training
Venue: arXiv cs.CR 2021
Year: 2021

Cybersecurity incident response teams mitigate the impact of adverse cyber-related events in organisations. Field studies of IR teams suggest that at present the process of IR is under-developed with a focus on the technological dimension with little consideration of practice capability. To address this gap, we develop a scenario-based training approach to assist organisations to overcome socio-technical barriers to incident response. The training approach is informed by a comprehensive list of socio-technical barriers compiled from a comprehensive review of the literature. Our primary contribution is a novel meta-level framework to generate scenarios specifically targeting socio-technical issues. To demonstrate the utility of the framework, a proof-of-concept scenario is presented.
A Transformer-based Math Language Model for Handwritten Math Expression Recognition
Venue: arXiv cs.CL 2021
Year: 2021

Handwritten mathematical expressions (HMEs) contain ambiguities in their interpretations, even for humans sometimes. Several math symbols are very similar in the writing style, such as dot and comma or 0, O, and o, which is a challenge for HME recognition systems to handle without using contextual information. To address this problem, this paper presents a Transformer-based Math Language Model (TMLM). Based on the self-attention mechanism, the high-level representation of an input token in a sequence of tokens is computed by how it is related to the previous tokens. Thus, TMLM can capture long dependencies and correlations among symbols and relations in a mathematical expression (ME). We trained the proposed language model using a corpus of approximately 70,000 LaTeX sequences provided in CROHME 2016. TMLM achieved the perplexity of 4.42, which outperformed the previous math language models, i.e., the N-gram and recurrent neural network-based language models. In addition, we combine TMLM into a stochastic context-free grammar-based HME recognition system using a weighting parameter to re-rank the top-10 best candidates. The expression rates on the testing sets of CROHME 2016 and CROHME 2019 were improved by 2.97 and 0.83 percentage points, respectively.
Learning Oculomotor Behaviors from Scanpath
Venue: arXiv cs.CV 2021
Year: 2021

Identifying oculomotor behaviors relevant for eye-tracking applications is a critical but often challenging task. Aiming to automatically learn and extract knowledge from existing eye-tracking data, we develop a novel method that creates rich representations of oculomotor scanpaths to facilitate the learning of downstream tasks. The proposed stimulus-agnostic Oculomotor Behavior Framework (OBF) model learns human oculomotor behaviors from unsupervised and semi-supervised tasks, including reconstruction, predictive coding, fixation identification, and contrastive learning tasks. The resultant pre-trained OBF model can be used in a variety of applications. Our pre-trained model outperforms baseline approaches and traditional scanpath methods in autism spectrum disorder and viewed-stimulus classification tasks. Ablation experiments further show our proposed method could achieve even better results with larger model sizes and more diverse eye-tracking training datasets, supporting the model's potential for future eye-tracking applications. Open source code: http://github.com/BeibinLi/OBF.
DEMix Layers: Disentangling Domains for Modular Language Modeling
Venue: arXiv cs.CL 2021
Year: 2021

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.
Road Mapping and Localization using Sparse Semantic Visual Features
Venue: arXiv cs.RO 2021
Year: 2021

We present a novel method for visual mapping and localization for autonomous vehicles, by extracting, modeling, and optimizing semantic road elements. Specifically, our method integrates cascaded deep models to detect standardized road elements instead of traditional point features, to seek for improved pose accuracy and map representation compactness. To utilize the structural features, we model road lights and signs by their representative deep keypoints for skeleton and boundary, and parameterize lanes via piecewise cubic splines. Based on the road semantic features, we build a complete pipeline for mapping and localization, which includes a) image processing front-end, b) sensor fusion strategies, and c) optimization backend. Experiments on public datasets and our testing platform have demonstrated the effectiveness and advantages of our method by outperforming traditional approaches.
Variable-Length Music Score Infilling via XLNet and Musically Specialized Positional Encoding
Venue: arXiv cs.SD 2021
Year: 2021

This paper proposes a new self-attention based model for music score infilling, i.e., to generate a polyphonic music sequence that fills in the gap between given past and future contexts. While existing approaches can only fill in a short segment with a fixed number of notes, or a fixed time span between the past and future contexts, our model can infill a variable number of notes (up to 128) for different time spans. We achieve so with three major technical contributions. First, we adapt XLNet, an autoregressive model originally proposed for unsupervised model pre-training, to music score infilling. Second, we propose a new, musically specialized positional encoding called relative bar encoding that better informs the model of notes' position within the past and future context. Third, to capitalize relative bar encoding, we perform look-ahead onset prediction to predict the onset of a note one time step before predicting the other attributes of the note. We compare our proposed model with two strong baselines and show that our model is superior in both objective and subjective analyses.
Position-based Contributive Embeddings for Aspect-Based Sentiment Analysis
Venue: arXiv cs.CL 2021
Year: 2021

Aspect-based sentiment analysis (ABSA), exploring sentiment polarity of aspect-given sentence, is a fine-grained task in the field of nature language processing. Previously researches typically tend to predict polarity based on the meaning of aspect and opinions. However, those approaches mainly focus on considering relations implicitly at the word level, ignore the historical impact of other positional words when the aspect appears in a certain position. Therefore, we propose a Position-based Contributive Embeddings (PosCE) to highlight the historical reference to special position aspect. Contribution of each positional words to the polarity is similar to the process of fairly distributing gains to several actors working in coalition (game theory). Therefore, we quote from the method of Shapley Value and finally gain PosCE to enhance the aspect-based representation for ABSA task. Furthermore, the PosCE can also be used for improving performances on multimodal ABSA task. Extensive experiments on both text and text-audio level using SemEval dataset show that the mainstream models advance performance in accuracy and F1 (increase 2.82% and 4.21% on average respectively) by applying our PosCE.