OpenRaiser/Intern-Atlas huggingface dataset:
CAMRI Loss: Improving Recall of a Specific Class without Sacrificing Accuracy
Venue: arXiv cs.LG 2022
Year: 2022

In real-world applications of multi-class classification models, misclassification in an important class (e.g., stop sign) can be significantly more harmful than in other classes (e.g., speed limit). In this paper, we propose a loss function that can improve the recall of an important class while maintaining the same level of accuracy as the case using cross-entropy loss. For our purpose, we need to make the separation of the important class better than the other classes. However, existing methods that give a class-sensitive penalty for cross-entropy loss do not improve the separation. On the other hand, the method that gives a margin to the angle between the feature vectors and the weight vectors of the last fully connected layer corresponding to each feature can improve the separation. Therefore, we propose a loss function that can improve the separation of the important class by setting the margin only for the important class, called Class-sensitive Additive Angular Margin Loss (CAMRI Loss). CAMRI loss is expected to reduce the variance of angles between features and weights of the important class relative to other classes due to the margin around the important class in the feature space by adding a penalty to the angle. In addition, concentrating the penalty only on the important classes hardly sacrifices the separation of the other classes. Experiments on CIFAR-10, GTSRB, and AwA2 showed that the proposed method could improve up to 9% recall improvement on cross-entropy loss without sacrificing accuracy.
Learning to Write with Coherence From Negative Examples
Venue: arXiv cs.CL 2022
Year: 2022

Coherence is one of the critical factors that determine the quality of writing. We propose writing relevance (WR) training method for neural encoder-decoder natural language generation (NLG) models which improves coherence of the continuation by leveraging negative examples. WR loss regresses the vector representation of the context and generated sentence toward positive continuation by contrasting it with the negatives. We compare our approach with Unlikelihood (UL) training in a text continuation task on commonsense natural language inference (NLI) corpora to show which method better models the coherence by avoiding unlikely continuations. The preference of our approach in human evaluation shows the efficacy of our method in improving coherence.
Software Product Line for Metaverse: Preliminary Results
Venue: arXiv cs.SE 2022
Year: 2022

The Metaverse is a network of eXtended Reality applications (XR apps) connected to each other, over the Internet infrastructure, allowing network users, systems, and devices to access them. It is very challenging to implement solutions for XR apps, due to the combination of complex concerns that should be addressed: multiple users with non-traditional input and output devices, different hardware platforms that should be addressed, forceful interactive rates, and experimental interaction techniques, among other issues. Therefore, this work aims to present a Software Product Line (SPL)-based approach to support the development of Web XR apps. More specifically, we define a features model that represents similarities and variables (domain analysis); we defined a core composed of generic and reusable software artifacts (domain project); and we developed an interface to support the instantiation of a Web XR app family, named MetaSee Features Model Editor (domain implementation). This approach integrates with a component of the MetaSEE architecture (Metaverse for Software Engineering Education). A preliminary assessment found that Features Model has conceptual consistency from the point of view of the complexity of Web XR Apps multimodal interaction. As future work, features model and artifacts will be increased with improvements and an evaluation with a significant number of participants will be made.
Privacy Attacks Against Biometric Models with Fewer Samples: Incorporating the Output of Multiple Models
Venue: arXiv cs.CV 2022
Year: 2022

Authentication systems are vulnerable to model inversion attacks where an adversary is able to approximate the inverse of a target machine learning model. Biometric models are a prime candidate for this type of attack. This is because inverting a biometric model allows the attacker to produce a realistic biometric input to spoof biometric authentication systems. One of the main constraints in conducting a successful model inversion attack is the amount of training data required. In this work, we focus on iris and facial biometric systems and propose a new technique that drastically reduces the amount of training data necessary. By leveraging the output of multiple models, we are able to conduct model inversion attacks with 1/10th the training set size of Ahmad and Fuller (IJCB 2020) for iris data and 1/1000th the training set size of Mai et al. (Pattern Analysis and Machine Intelligence 2019) for facial data. We denote our new attack technique as structured random with alignment loss. Our attacks are black-box, requiring no knowledge of the weights of the target neural network, only the dimension, and values of the output vector. To show the versatility of the alignment loss, we apply our attack framework to the task of membership inference (Shokri et al., IEEE S&P 2017) on biometric data. For the iris, membership inference attack against classification networks improves from 52% to 62% accuracy.
DeepThought: a Reputation and Voting-based Blockchain Oracle
Venue: arXiv cs.CR 2022
Year: 2022

Thanks to built-in immutability and persistence, the blockchain is often seen as a promising technology to certify information. However, when the information does not originate from the blockchain itself, its correctness cannot be taken for granted. To address this limitation, blockchain oracles -- services that validate external information before storing it in a blockchain -- were introduced. In particular, when the validation cannot be automated, oracles rely on humans that collaboratively cross-check external information. In this paper, we present DeepThought, a distributed human-based oracle that combines voting and reputation schemes. An empirical evaluation compares DeepThought with a state-of-the-art solution and shows that our approach achieves greater resistance to voters corruptions in different configurations.
Outage Probability Analysis of RF/FSO-VLC Communication Relaying System
Venue: arXiv cs.NI 2022
Year: 2022

This paper presents an analysis of the asymmetric relaying system which provides communication between hybrid outdoor sub-system and indoor visible light communications (VLC) access points. The outdoor sub-system represents a hybrid radio-frequency (RF)/free-space optical (FSO) system, introduced to reduce the impact of weather conditions on transmission quality. Closed-form outage probability analytical expression is derived. Numerical results are presented and confirmed by Monte Carlo simulations. The effects of system and channel parameters on the outage probability performance are investigated and discussed. Greater optical transmitted power of the VLC subsystem reflects in better system performance, as well as lower indoor environment height. When the indoor room is higher, the propagation of the optical signal is longer, and there will be greater power dissipation and performance deterioration. In addition, the outage probability floor is noticed, which is important limiting factor for relaying system design.
Designing an Automated Vehicle: Strategies for Handling Tasks of a Previously Required Accompanying Person
Venue: arXiv cs.HC 2022
Year: 2022

When using a conventional passenger car, several groups of people are reliant on the assistance of an accompanying person, for example when getting in and out of the car. For the independent use of an automatically driving vehicle by those groups, the absence of a previously required accompanying person needs to be compensated. During the design process of an autonomous family vehicle, we found that a low-barrier vehicle design can only partly contribute to the compensation for the absence of a required human companion. In this paper, we present four strategies we identified for handling the tasks of a previously required accompanying individual. The presented top-down approach supports developers in identifying unresolved problems, in finding, structuring, and selecting solutions as well as in uncovering upcoming problems at an early stage in the development of novel concepts for driverless vehicles. As an example, we consider the hypothetical exit of persons in need of assistance. The application of the four strategies in this example demonstrates the far-reaching impact of consistently considering users in need of support in the development of automated vehicles.
Query-based Industrial Analytics over Knowledge Graphs with Ontology Reshaping
Venue: arXiv cs.DB 2022
Year: 2022

Industrial analytics that includes among others equipment diagnosis and anomaly detection heavily relies on integration of heterogeneous production data. Knowledge Graphs (KGs) as the data format and ontologies as the unified data schemata are a prominent solution that offers high quality data integration and a convenient and standardised way to exchange data and to layer analytical applications over it. However, poor design of ontologies of high degree of mismatch between them and industrial data naturally lead to KGs of low quality that impede the adoption and scalability of industrial analytics. Indeed, such KGs substantially increase the training time of writing queries for users, consume high volume of storage for redundant information, and are hard to maintain and update. To address this problem we propose an ontology reshaping approach to transform ontologies into KG schemata that better reflect the underlying data and thus help to construct better KGs. In this poster we present a preliminary discussion of our on-going research, evaluate our approach with a rich set of SPARQL queries on real-world industry data at Bosch and discuss our findings.
This is what a pandemic looks like: Visual framing of COVID-19 on search engines
Venue: arXiv cs.IR 2022
Year: 2022

In today's high-choice media environment, search engines play an integral role in informing individuals and societies about the latest events. The importance of search algorithms is even higher at the time of crisis, when users search for information to understand the causes and the consequences of the current situation and decide on their course of action. In our paper, we conduct a comparative audit of how different search engines prioritize visual information related to COVID-19 and what consequences it has for the representation of the pandemic. Using a virtual agent-based audit approach, we examine image search results for the term "coronavirus" in English, Russian and Chinese on five major search engines: Google, Yandex, Bing, Yahoo, and DuckDuckGo. Specifically, we focus on how image search results relate to generic news frames (e.g., the attribution of responsibility, human interest, and economics) used in relation to COVID-19 and how their visual composition varies between the search engines.
Modern Machine Learning Tools for Monitoring and Control of Industrial Processes: A Survey
Venue: arXiv cs.LG 2022
Year: 2022

Over the last ten years, we have seen a significant increase in industrial data, tremendous improvement in computational power, and major theoretical advances in machine learning. This opens up an opportunity to use modern machine learning tools on large-scale nonlinear monitoring and control problems. This article provides a survey of recent results with applications in the process industry.
Perceived Security of E-Learning Portal
Venue: arXiv cs.CR 2022
Year: 2022

Information technology has made e-learning possible and available on a large scale. Learning management system (LMS) has been widely used and is accessible through the Internet. However, LMS are exposed to various threats. Proper understanding of the threats is required. Furthermore strategy and best practices countermeasures will ensure a safe learning environment. Therefore, this study looks into the information security aspect of LMS. Specifically, there are two main purposes of this study. First, this study provides a review of information security in e-learning environments and explains the importance of information security. Second, this study looks at how student perceived the security of their e-learning portal. A total of 497 students responded to a survey questionnaires. Frequencies analysis was conducted to show the profile of the respondent. Overall, respondent has strong positive perceptions towards security of their LMS. This study serve as an introduction which help LMS administrator to understand the issues and possibilities related to the safety of LMS.
NamedMask: Distilling Segmenters from Complementary Foundation Models
Venue: arXiv cs.CV 2022
Year: 2022

The goal of this work is to segment and name regions of images without access to pixel-level labels during training. To tackle this task, we construct segmenters by distilling the complementary strengths of two foundation models. The first, CLIP (Radford et al. 2021), exhibits the ability to assign names to image content but lacks an accessible representation of object structure. The second, DINO (Caron et al. 2021), captures the spatial extent of objects but has no knowledge of object names. Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images. These images are pseudo-labelled with a category-agnostic salient object detector bootstrapped from DINO, then refined by category-specific segmenters using the CLIP archive labels. Thanks to the high quality of the refined masks, we show that a standard segmentation architecture trained on these archives with appropriate data augmentation achieves impressive semantic segmentation abilities for both single-object and multi-object images. As a result, our proposed NamedMask performs favourably against a range of prior work on five benchmarks including the VOC2012, COCO and large-scale ImageNet-S datasets.
Capsule Network based Contrastive Learning of Unsupervised Visual Representations
Venue: arXiv cs.CV 2022
Year: 2022

Capsule Networks have shown tremendous advancement in the past decade, outperforming the traditional CNNs in various task due to it's equivariant properties. With the use of vector I/O which provides information of both magnitude and direction of an object or it's part, there lies an enormous possibility of using Capsule Networks in unsupervised learning environment for visual representation tasks such as multi class image classification. In this paper, we propose Contrastive Capsule (CoCa) Model which is a Siamese style Capsule Network using Contrastive loss with our novel architecture, training and testing algorithm. We evaluate the model on unsupervised image classification CIFAR-10 dataset and achieve a top-1 test accuracy of 70.50% and top-5 test accuracy of 98.10%. Due to our efficient architecture our model has 31 times less parameters and 71 times less FLOPs than the current SOTA in both supervised and unsupervised learning.
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Venue: arXiv cs.RO 2022
Year: 2022

Task planning can require defining myriad domain knowledge about the world in which a robot needs to act. To ameliorate that effort, large language models (LLMs) can be used to score potential next actions during task planning, and even generate action sequences directly, given an instruction in natural language with no additional domain information. However, such methods either require enumerating all possible next steps for scoring, or generate free-form text that may contain actions not possible on a given robot in its current context. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments, robot capabilities, and tasks. Our key insight is to prompt the LLM with program-like specifications of the available actions and objects in an environment, as well as with example programs that can be executed. We make concrete recommendations about prompt structure and generation constraints through ablation experiments, demonstrate state of the art success rates in VirtualHome household tasks, and deploy our method on a physical robot arm for tabletop tasks. Website at progprompt.github.io
Spatial model personalization in Gboard
Venue: arXiv cs.HC 2022
Year: 2022

We introduce a framework for adapting a virtual keyboard to individual user behavior by modifying a Gaussian spatial model to use personalized key center offset means and, optionally, learned covariances. Through numerous real-world studies, we determine the importance of training data quantity and weights, as well as the number of clusters into which to group keys to avoid overfitting. While past research has shown potential of this technique using artificially-simple virtual keyboards and games or fixed typing prompts, we demonstrate effectiveness using the highly-tuned Gboard app with a representative set of users and their real typing behaviors. Across a variety of top languages, we achieve small-but-significant improvements in both typing speed and decoder accuracy.
Learning Certifiably Robust Controllers Using Fragile Perception
Venue: arXiv cs.RO 2022
Year: 2022

Advances in computer vision and machine learning enable robots to perceive their surroundings in powerful new ways, but these perception modules have well-known fragilities. We consider the problem of synthesizing a safe controller that is robust despite perception errors. The proposed method constructs a state estimator based on Gaussian processes with input-dependent noises. This estimator computes a high-confidence set for the actual state given a perceived state. Then, a robust neural network controller is synthesized that can provably handle the state uncertainty. Furthermore, an adaptive sampling algorithm is proposed to jointly improve the estimator and controller. Simulation experiments, including a realistic vision-based lane-keeping example in CARLA, illustrate the promise of the proposed approach in synthesizing robust controllers with deep-learning-based perception.
Enabling Homogeneous GNNs to Handle Heterogeneous Graphs via Relation Embedding
Venue: arXiv cs.LG 2022
Year: 2022

Graph Neural Networks (GNNs) have been generalized to process the heterogeneous graphs by various approaches. Unfortunately, these approaches usually model the heterogeneity via various complicated modules. This paper aims to propose a simple yet effective framework to assign adequate ability to the homogeneous GNNs to handle the heterogeneous graphs. Specifically, we propose Relation Embedding based Graph Neural Network (RE-GNN), which employs only one parameter per relation to embed the importance of distinct types of relations and node-type-specific self-loop connections. To optimize these relation embeddings and the model parameters simultaneously, a gradient scaling factor is proposed to constrain the embeddings to converge to suitable values. Besides, we interpret the proposed RE-GNN from two perspectives, and theoretically demonstrate that our RE-GCN possesses more expressive power than GTN (which is a typical heterogeneous GNN, and it can generate meta-paths adaptively). Extensive experiments demonstrate that our RE-GNN can effectively and efficiently handle the heterogeneous graphs and can be applied to various homogeneous GNNs.
Modeling and Leveraging Prerequisite Context in Recommendation
Venue: arXiv cs.IR 2022
Year: 2022

Prerequisites can play a crucial role in users' decision-making yet recommendation systems have not fully utilized such contextual background knowledge. Traditional recommendation systems (RS) mostly enrich user-item interactions where the context consists of static user profiles and item descriptions, ignoring the contextual logic and constraints that underlie them. For example, an RS may recommend an item on the condition that the user has interacted with another item as its prerequisite. Modeling prerequisite context from conceptual side information can overcome this weakness. We propose Prerequisite Driven Recommendation (PDR), a generic context-aware framework where prerequisite context is explicitly modeled to facilitate recommendation. We first design a Prerequisite Knowledge Linking (PKL) algorithm, to curate datasets facilitating PDR research. Employing it, we build a 75k+ high-quality prerequisite concept dataset which spans three domains. We then contribute PDRS, a neural instantiation of PDR. By jointly optimizing both the prerequisite learning and recommendation tasks through multi-layer perceptrons, we find PDRS consistently outperforms baseline models in all three domains, by an average margin of 7.41%. Importantly, PDRS performs especially well in cold-start scenarios with improvements of up to 17.65%.
AvatarGo: Plug and Play self-avatars for VR
Venue: arXiv cs.HC 2022
Year: 2022

The use of self-avatars in a VR application can enhance presence and embodiment which leads to a better user experience. In collaborative VR it also facilitates non-verbal communication. Currently it is possible to track a few body parts with cheap trackers and then apply IK methods to animate a character. However, the correspondence between trackers and avatar joints is typically fixed ad-hoc, which is enough to animate the avatar, but causes noticeable mismatches between the user's body pose and the avatar. In this paper we present a fast and easy to set up system to compute exact offset values, unique for each user, which leads to improvements in avatar movement. Our user study shows that the Sense of Embodiment increased significantly when using exact offsets as opposed to fixed ones. We also allowed the users to see a semitransparent avatar overlaid with their real body to objectively evaluate the quality of the avatar movement with our technique.
Time Series Causal Link Estimation under Hidden Confounding using Knockoff Interventions
Venue: arXiv cs.LG 2022
Year: 2022

Latent variables often mask cause-effect relationships in observational data which provokes spurious links that may be misinterpreted as causal. This problem sparks great interest in the fields such as climate science and economics. We propose to estimate confounded causal links of time series using Sequential Causal Effect Variational Autoencoder (SCEVAE) while applying Knockoff interventions. Knockoff variables have the same distribution as the originals and preserve the correlation to other variables. This allows for counterfactuals that are more faithful to the observational distribution. We show the advantage of Knockoff interventions by applying SCEVAE to synthetic datasets with both linear and nonlinear causal links. Moreover, we apply SCEVAE with Knockoffs to real aerosol-cloud-climate observational time series data. We compare our results on synthetic data to those of a time series deconfounding method both with and without estimated confounders. We show that our method outperforms this benchmark by comparing both methods to the ground truth. For the real data analysis, we rely on expert knowledge of causal links and demonstrate how using suitable proxy variables improves the causal link estimation in the presence of hidden confounders.