OpenRaiser/Intern-Atlas huggingface dataset:
Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders
Venue: arXiv cs.CL 2021
Year: 2021

The growing deluge of scientific publications demands text analysis tools that can help scientists and policy-makers navigate, forecast and beneficially guide scientific research. Recent advances in natural language understanding driven by deep transformer networks offer new possibilities for mapping science. Because the same surface text can take on multiple and sometimes contradictory specialized senses across distinct research communities, sensitivity to context is critical for infometric applications. Transformer embedding models such as BERT capture shades of association and connotation that vary across the different linguistic contexts of any particular word or span of text. Here we report a procedure for encoding scientific documents with these tools, measuring their improvement over static word embeddings in a nearest-neighbor retrieval task. We find discriminability of contextual representations is strongly influenced by choice of pooling strategy for summarizing the high-dimensional network activations. Importantly, we note that fundamentals such as domain-matched training data are more important than state-of-the-art NLP tools. Yet state-of-the-art models did offer significant gains. The best approach we investigated combined domain-matched pretraining, sound pooling, and state-of-the-art deep transformer network encoders. Finally, with the goal of leveraging contextual representations from deep encoders, we present a range of measurements for understanding and forecasting research communities in science.
Learning Multi-modal Information for Robust Light Field Depth Estimation
Venue: arXiv cs.CV 2021
Year: 2021

Light field data has been demonstrated to facilitate the depth estimation task. Most learning-based methods estimate the depth infor-mation from EPI or sub-aperture images, while less methods pay attention to the focal stack. Existing learning-based depth estimation methods from the focal stack lead to suboptimal performance because of the defocus blur. In this paper, we propose a multi-modal learning method for robust light field depth estimation. We first excavate the internal spatial correlation by designing a context reasoning unit which separately extracts comprehensive contextual information from the focal stack and RGB images. Then we integrate the contextual information by exploiting a attention-guide cross-modal fusion module. Extensive experiments demonstrate that our method achieves superior performance than existing representative methods on two light field datasets. Moreover, visual results on a mobile phone dataset show that our method can be widely used in daily life.
Joint Optimization of Preamble Selection and Access Barring for MTC with Correlated Device Activities
Venue: arXiv cs.IT 2021
Year: 2021

Most existing works on random access for machine-type communication (MTC) assume independent device activities. However, in several Internet-of-Things (IoT) applications, device activities are driven by events and hence may be correlated. This paper investigates the joint optimization of preamble selection and access barring for correlated device activities. We adopt a random access scheme with general random preamble selection parameterized by the preamble selection distributions of all devices and an access barring scheme parameterized by the access barring factor, to maximally exploit correlated device activities for improving the average throughput. First, we formulate the average throughput maximization problem with respect to the preamble selection distributions and the access barring factor. It is a challenging nonconvex problem. We characterize an optimality property of the problem. Then, we develop two iterative algorithms to obtain a stationary point and a low-complexity solution respectively by using the block coordinate descend (BCD) method. Numerical results show that the two proposed solutions achieve significant gains over existing schemes, demonstrating the significance of exploiting correlation of device activities in improving the average throughput. Numerical results also show that compared to the stationary point, the low-complexity solution achieves a similar average throughput with much lower computational complexity, demonstrating the effectiveness of the low-complexity solution.
SPARK: SPAcecraft Recognition leveraging Knowledge of Space Environment
Venue: arXiv cs.CV 2021
Year: 2021

This paper proposes the SPARK dataset as a new unique space object multi-modal image dataset. Image-based object recognition is an important component of Space Situational Awareness, especially for applications such as on-orbit servicing, active debris removal, and satellite formation. However, the lack of sufficient annotated space data has limited research efforts in developing data-driven spacecraft recognition approaches. The SPARK dataset has been generated under a realistic space simulation environment, with a large diversity in sensing conditions for different orbital scenarios. It provides about 150k images per modality, RGB and depth, and 11 classes for spacecrafts and debris. This dataset offers an opportunity to benchmark and further develop object recognition, classification and detection algorithms, as well as multi-modal RGB-Depth approaches under space sensing conditions. Preliminary experimental evaluation validates the relevance of the data, and highlights interesting challenging scenarios specific to the space environment.
Detecting Operational Adversarial Examples for Reliable Deep Learning
Venue: arXiv cs.SE 2021
Year: 2021

The utilisation of Deep Learning (DL) raises new challenges regarding its dependability in critical applications. Sound verification and validation methods are needed to assure the safe and reliable use of DL. However, state-of-the-art debug testing methods on DL that aim at detecting adversarial examples (AEs) ignore the operational profile, which statistically depicts the software's future operational use. This may lead to very modest effectiveness on improving the software's delivered reliability, as the testing budget is likely to be wasted on detecting AEs that are unrealistic or encountered very rarely in real-life operation. In this paper, we first present the novel notion of "operational AEs" which are AEs that have relatively high chance to be seen in future operation. Then an initial design of a new DL testing method to efficiently detect "operational AEs" is provided, as well as some insights on our prospective research plan.
Reproducible Builds: Increasing the Integrity of Software Supply Chains
Venue: arXiv cs.SE 2021
Year: 2021

Although it is possible to increase confidence in Free and Open Source Software (FOSS) by reviewing its source code, trusting code is not the same as trusting its executable counterparts. These are typically built and distributed by third-party vendors, with severe security consequences if their supply chains are compromised. In this paper, we present reproducible builds, an approach that can determine whether generated binaries correspond with their original source code. We first define the problem, and then provide insight into the challenges of making real-world software build in a "reproducible" manner-this is, when every build generates bit-for-bit identical results. Through the experience of the Reproducible Builds project making the Debian Linux distribution reproducible, we also describe the affinity between reproducibility and quality assurance (QA).
Conclusive Local Interpretation Rules for Random Forests
Venue: arXiv cs.LG 2021
Year: 2021

In critical situations involving discrimination, gender inequality, economic damage, and even the possibility of casualties, machine learning models must be able to provide clear interpretations for their decisions. Otherwise, their obscure decision-making processes can lead to socioethical issues as they interfere with people's lives. In the aforementioned sectors, random forest algorithms strive, thus their ability to explain themselves is an obvious requirement. In this paper, we present LionForests, which relies on a preliminary work of ours. LionForests is a random forest-specific interpretation technique, which provides rules as explanations. It is applicable from binary classification tasks to multi-class classification and regression tasks, and it is supported by a stable theoretical background. Experimentation, including sensitivity analysis and comparison with state-of-the-art techniques, is also performed to demonstrate the efficacy of our contribution. Finally, we highlight a unique property of LionForests, called conclusiveness, that provides interpretation validity and distinguishes it from previous techniques.
Transformer-based Methods for Recognizing Ultra Fine-grained Entities (RUFES)
Venue: arXiv cs.CL 2021
Year: 2021

This paper summarizes the participation of the Laboratoire Informatique, Image et Interaction (L3i laboratory) of the University of La Rochelle in the Recognizing Ultra Fine-grained Entities (RUFES) track within the Text Analysis Conference (TAC) series of evaluation workshops. Our participation relies on two neural-based models, one based on a pre-trained and fine-tuned language model with a stack of Transformer layers for fine-grained entity extraction and one out-of-the-box model for within-document entity coreference. We observe that our approach has great potential in increasing the performance of fine-grained entity recognition. Thus, the future work envisioned is to enhance the ability of the models following additional experiments and a deeper analysis of the results.
Group Recommendation Techniques for Feature Modeling and Configuration
Venue: arXiv cs.AI 2021
Year: 2021

In large-scale feature models, feature modeling and configuration processes are highly expected to be done by a group of stakeholders. In this context, recommendation techniques can increase the efficiency of feature-model design and find optimal configurations for groups of stakeholders. Existing studies show plenty of issues concerning feature model navigation support, group members' satisfaction, and conflict resolution. This study proposes group recommendation techniques for feature modeling and configuration on the basis of addressing the mentioned issues.
Adaptive Logit Adjustment Loss for Long-Tailed Visual Recognition
Venue: arXiv cs.CV 2021
Year: 2021

Data in the real world tends to exhibit a long-tailed label distribution, which poses great challenges for the training of neural networks in visual recognition. Existing methods tackle this problem mainly from the perspective of data quantity, i.e., the number of samples in each class. To be specific, they pay more attention to tail classes, like applying larger adjustments to the logit. However, in the training process, the quantity and difficulty of data are two intertwined and equally crucial problems. For some tail classes, the features of their instances are distinct and discriminative, which can also bring satisfactory accuracy; for some head classes, although with sufficient samples, the high semantic similarity with other classes and lack of discriminative features will bring bad accuracy. Based on these observations, we propose Adaptive Logit Adjustment Loss (ALA Loss) to apply an adaptive adjusting term to the logit. The adaptive adjusting term is composed of two complementary factors: 1) quantity factor, which pays more attention to tail classes, and 2) difficulty factor, which adaptively pays more attention to hard instances in the training process. The difficulty factor can alleviate the over-optimization on tail yet easy instances and under-optimization on head yet hard instances. The synergy of the two factors can not only advance the performance on tail classes even further, but also promote the accuracy on head classes. Unlike previous logit adjusting methods that only concerned about data quantity, ALA Loss tackles the long-tailed problem from a more comprehensive, fine-grained and adaptive perspective. Extensive experimental results show that our method achieves the state-of-the-art performance on challenging recognition benchmarks, including ImageNet-LT, iNaturalist 2018, and Places-LT.
Relevance-Aware Anomalous Users Detection in Social Network via Graph Neural Network
Venue: arXiv cs.SI 2021
Year: 2021

Anomalous users detection in social network is an imperative task for security problems. Motivated by the great power of Graph Neural Networks(GNNs), many current researches adopt GNN-based detectors to reveal the anomalous users. However, the increasing scale of social activities, explosive growth of users and manifold technical disguise render the user detection a difficult task. In this paper, we propose an innovate Relevance-aware Anomalous Users Detection model (RAU-GNN) to obtain a fine-grained detection result. RAU-GNN first extracts multiple relations of all types of users in social network, including both benign and anomalous users, and accordingly constructs the multiple user relation graph. Secondly, we employ relevance-aware GNN framework to learn the hidden features of users, and discriminate the anomalous users after discriminating. Concretely, by integrating Graph Convolution Network(GCN) and Graph Attention Network(GAT), we design a GCN-based relation fusion layer to aggregate initial information from different relations, and a GAT-based embedding layer to obtain the high-level embeddings. Lastly, we feed the learned representations to the following GNN layer in order to consolidate the node embedding by aggregating the final users' embeddings. We conduct extensive experiment on real-world datasets. The experimental results show that our approach can achieve high accuracy for anomalous users detection.
Optimizing the Long-Term Average Reward for Continuing MDPs: A Technical Report
Venue: arXiv cs.LG 2021
Year: 2021

Recently, we have struck the balance between the information freshness, in terms of age of information (AoI), experienced by users and energy consumed by sensors, by appropriately activating sensors to update their current status in caching enabled Internet of Things (IoT) networks [1]. To solve this problem, we cast the corresponding status update procedure as a continuing Markov Decision Process (MDP) (i.e., without termination states), where the number of state-action pairs increases exponentially with respect to the number of considered sensors and users. Moreover, to circumvent the curse of dimensionality, we have established a methodology for designing deep reinforcement learning (DRL) algorithms to maximize (resp. minimize) the average reward (resp. cost), by integrating R-learning, a tabular reinforcement learning (RL) algorithm tailored for maximizing the long-term average reward, and traditional DRL algorithms, initially developed to optimize the discounted long-term cumulative reward rather than the average one. In this technical report, we would present detailed discussions on the technical contributions of this methodology.
Probabilistic Accumulate-then-Transmit in Wireless-Powered Covert Communications
Venue: arXiv cs.IT 2021
Year: 2021

In this paper, we investigate the optimal design of a wireless-powered covert communication (WP-CC) system, in which a probabilistic accumulate-then-transmit (ATT) protocol is proposed to maximize the communication covertness subject to a quality-of-service (QoS) requirement on communication. Specifically, in the considered WP-CC system, a full-duplex (FD) receiver transmits artificial noise (AN) to simultaneously charge an energy-constrained transmitter and to confuse a warden's detection on the transmitter's communication activity. With the probabilistic ATT protocol, the transmitter sends its information with a prior probability, i.e., $p$, conditioned on the available energy being sufficient. Our analysis shows that the probabilistic ATT protocol can achieve higher covertness than the traditional ATT protocol with $p=1$. In order to facilitate the optimal design of the WP-CC system, we also derive the warden's minimum detection error probability and characterize the effective covert rate from the transmitter to the receiver to quantify the communication covertness and quality, respectively. The derived analytical results facilitate the joint optimization of the probability $p$ and the information transmit power. We further present the optimal design of a cable-powered covert communication (CP-CC) system as a benchmark for comparison. Our simulation shows that the proposed probabilistic ATT protocol (with a varying $p$) can achieve the covertness upper bound determined by the CP-CC system, while the traditional ATT protocol (with $p=1$) cannot, which again confirms the benefits brought by the proposed probabilistic ATT in covert communications.
Terrain assessment for precision agriculture using vehicle dynamic modelling
Venue: arXiv cs.RO 2021
Year: 2021

Advances in precision agriculture greatly rely on innovative control and sensing technologies that allow service units to increase their level of driving automation while ensuring at the same time high safety standards. This paper deals with automatic terrain estimation and classification that is performed simultaneously by an agricultural vehicle during normal operations. Vehicle mobility and safety, and the successful implementation of important agricultural tasks including seeding, ploughing, fertilising and controlled traffic depend or can be improved by a correct identification of the terrain that is traversed. The novelty of this research lies in that terrain estimation is performed by using not only traditional appearance-based features, that is colour and geometric properties, but also contact-based features, that is measuring physics-based dynamic effects that govern the vehicleeterrain interaction and that greatly affect its mobility. Experimental results obtained from an all-terrain vehicle operating on different surfaces are presented to validate the system in the field. It was shown that a terrain classifier trained with contact features was able to achieve a correct prediction rate of 85.1%, which is comparable or better than that obtained with approaches using traditional feature sets. To further improve the classification performance, all feature sets were merged in an augmented feature space, reaching, for these tests, 89.1% of correct predictions.
Real-time Forecast Models for TBM Load Parameters Based on Machine Learning Methods
Venue: arXiv cs.LG 2021
Year: 2021

Because of the fast advance rate and the improved personnel safety, tunnel boring machines (TBMs) have been widely used in a variety of tunnel construction projects. The dynamic modeling of TBM load parameters (including torque, advance rate and thrust) plays an essential part in the design, safe operation and fault prognostics of this complex engineering system. In this paper, based on in-situ TBM operational data, we use the machine-learning (ML) methods to build the real-time forecast models for TBM load parameters, which can instantaneously provide the future values of the TBM load parameters as long as the current data are collected. To decrease the model complexity and improve the generalization, we also apply the least absolute shrinkage and selection (Lasso) method to extract the essential features of the forecast task. The experimental results show that the forecast models based on deep-learning methods, {\it e.g.}, recurrent neural network and its variants, outperform the ones based on the shallow-learning methods, {\it e.g.}, support vector regression and random forest. Moreover, the Lasso-based feature extraction significantly improves the performance of the resultant models.
Mediators in Determining what Processing BERT Performs First
Venue: arXiv cs.CL 2021
Year: 2021

Probing neural models for the ability to perform downstream tasks using their activation patterns is often used to localize what parts of the network specialize in performing what tasks. However, little work addressed potential mediating factors in such comparisons. As a test-case mediating factor, we consider the prediction's context length, namely the length of the span whose processing is minimally required to perform the prediction. We show that not controlling for context length may lead to contradictory conclusions as to the localization patterns of the network, depending on the distribution of the probing dataset. Indeed, when probing BERT with seven tasks, we find that it is possible to get 196 different rankings between them when manipulating the distribution of context lengths in the probing dataset. We conclude by presenting best practices for conducting such comparisons in the future.
Reward Shaping with Subgoals for Social Navigation
Venue: arXiv cs.RO 2021
Year: 2021

Social navigation has been gaining attentions with the growth in machine intelligence. Since reinforcement learning can select an action in the prediction phase at a low computational cost, it has been formulated in a social navigation tasks. However, reinforcement learning takes an enormous number of iterations until acquiring a behavior policy in the learning phase. This negatively affects the learning of robot behaviors in the real world. In particular, social navigation includes humans who are unpredictable moving obstacles in an environment. We proposed a reward shaping method with subgoals to accelerate learning. The main part is an aggregation method that use subgoals to shape a reinforcement learning algorithm. We performed a learning experiment with a social navigation task in which a robot avoided collisions and then reached its goal. The experimental results show that our method improved the learning efficiency from a base algorithm in the task.
Zhestyatsky at SemEval-2021 Task 2: ReLU over Cosine Similarity for BERT Fine-tuning
Venue: arXiv cs.CL 2021
Year: 2021

This paper presents our contribution to SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation (MCL-WiC). Our experiments cover English (EN-EN) sub-track from the multilingual setting of the task. We experiment with several pre-trained language models and investigate an impact of different top-layers on fine-tuning. We find the combination of Cosine Similarity and ReLU activation leading to the most effective fine-tuning procedure. Our best model results in accuracy 92.7%, which is the fourth-best score in EN-EN sub-track.
Research on Innovation in China and Latin America: bibliometric insights in business, management, accounting, and decision sciences
Venue: arXiv cs.DL 2021
Year: 2021

This study aims to comprehend the structure of RIBM (research on innovation in business and management) in China and LAC (Latin America and the Caribbean) via co-word and institutional co-authorship networks using Scopus' bibliographic data (1998- 2018). Multiple Correspondence Analysis and Social Network Analysis were applied. Public institutions are interconnected and generate most of the advances in RIBM. RIBM boards regional and national STi policies permeated by sustainability-related factors. China is focused on IT and knowledge management for supply chain and engineering, while LAC focuses on institutional perspectives for economic development.