OpenRaiser/Intern-Atlas huggingface dataset:
The Emergence of League and Sub-League Structure in the Population Lotto Game
Venue: arXiv cs.GT 2022
Year: 2022

In order to understand if and how strategic resource allocation can constrain the structure of pair-wise competition outcomes in competitive human competitions we introduce a new multiplayer resource allocation game, the Population Lotto Game. This new game allows agents to allocate their resources across a continuum of possible specializations. While this game allows non-transitive cycles between players, we show that the Nash equilibrium of the game also forms a hierarchical structure between discrete `leagues' based on their different resource budgets, with potential sub-league structure and/or non-transitive cycles inside individual leagues. We provide an algorithm that can find a particular Nash equilibrium for any finite set of discrete sub-population sizes and budgets. Further, our algorithm finds the unique Nash equilibrium that remains stable for the subset of players with budgets below any threshold.
Contextualized language models for semantic change detection: lessons learned
Venue: arXiv cs.CL 2022
Year: 2022

We present a qualitative analysis of the (potentially erroneous) outputs of contextualized embedding-based methods for detecting diachronic semantic change. First, we introduce an ensemble method outperforming previously described contextualized approaches. This method is used as a basis for an in-depth analysis of the degrees of semantic change predicted for English words across 5 decades. Our findings show that contextualized methods can often predict high change scores for words which are not undergoing any real diachronic semantic shift in the lexicographic sense of the term (or at least the status of these shifts is questionable). Such challenging cases are discussed in detail with examples, and their linguistic categorization is proposed. Our conclusion is that pre-trained contextualized language models are prone to confound changes in lexicographic senses and changes in contextual variance, which naturally stem from their distributional nature, but is different from the types of issues observed in methods based on static embeddings. Additionally, they often merge together syntactic and semantic aspects of lexical entities. We propose a range of possible future solutions to these issues.
Orloj: Predictably Serving Unpredictable DNNs
Venue: arXiv cs.DC 2022
Year: 2022

Existing DNN serving solutions can provide tight latency SLOs while maintaining high throughput via careful scheduling of incoming requests, whose execution times are assumed to be highly predictable and data-independent. However, inference requests to emerging dynamic DNNs -- e.g., popular natural language processing (NLP) models and computer vision (CV) models that skip layers -- are data-dependent. They exhibit poor performance when served using existing solutions because they experience large variance in request execution times depending on the input -- the longest request in a batch inflates the execution times of the smaller ones, causing SLO misses in the absence of careful batching. In this paper, we present Orloj, a dynamic DNN serving system, that captures this variance in dynamic DNNs using empirical distributions of expected request execution times, and then efficiently batches and schedules them without knowing a request's precise execution time. Orloj significantly outperforms state-of-the-art serving solutions for high variance dynamic DNN workloads by 51--80% in finish rate under tight SLO constraints, and over 100% under more relaxed SLO settings. For well-studied static DNN workloads, Orloj keeps comparable performance with the state-of-the-art.
1st Place Solution to ECCV 2022 Challenge on Out of Vocabulary Scene Text Understanding: End-to-End Recognition of Out of Vocabulary Words
Venue: arXiv cs.CV 2022
Year: 2022

Scene text recognition has attracted increasing interest in recent years due to its wide range of applications in multilingual translation, autonomous driving, etc. In this report, we describe our solution to the Out of Vocabulary Scene Text Understanding (OOV-ST) Challenge, which aims to extract out-of-vocabulary (OOV) words from natural scene images. Our oCLIP-based model achieves 28.59\% in h-mean which ranks 1st in end-to-end OOV word recognition track of OOV Challenge in ECCV2022 TiE Workshop.
Attack Tactic Identification by Transfer Learning of Language Model
Venue: arXiv cs.CR 2022
Year: 2022

Cybersecurity has become a primary global concern with the rapid increase in security attacks and data breaches. Artificial intelligence is promising to help humans analyzing and identifying attacks. However, labeling millions of packets for supervised learning is never easy. This study aims to leverage transfer learning technique that stores the knowledge gained from well-defined attack lifecycle documents and applies it to hundred thousands of unlabeled attacks (packets) for identifying their attack tactics. We anticipate the knowledge of an attack is well-described in the documents, and the cutting edge transformer-based language model can embed the knowledge into a high-dimensional latent space. Then, reusing the information from the language model for the learning of attack tactic carried by packets to improve the learning efficiency. We propose a system, PELAT, that fine-tunes BERT model with 1,417 articles from MITRE ATT&CK lifecycle framework to enhance its attack knowledge (including syntax used and semantic meanings embedded). PELAT then transfers its knowledge to perform semi-supervised learning for unlabeled packets to generate their tactic labels. Further, when a new attack packet arrives, the packet payload will be processed by the PELAT language model with a downstream classifier to predict its tactics. In this way, we can effectively reduce the burden of manually labeling big datasets. In a one-week honeypot attack dataset (227 thousand packets per day), PELAT performs 99% of precision, recall, and F1 on testing dataset. PELAT can infer over 99% of tactics on two other testing datasets (while nearly 90% of tactics are identified).
Video-Guided Curriculum Learning for Spoken Video Grounding
Venue: arXiv cs.CV 2022
Year: 2022

In this paper, we introduce a new task, spoken video grounding (SVG), which aims to localize the desired video fragments from spoken language descriptions. Compared with using text, employing audio requires the model to directly exploit the useful phonemes and syllables related to the video from raw speech. Moreover, we randomly add environmental noises to this speech audio, further increasing the difficulty of this task and better simulating real applications. To rectify the discriminative phonemes and extract video-related information from noisy audio, we develop a novel video-guided curriculum learning (VGCL) during the audio pre-training process, which can make use of the vital visual perceptions to help understand the spoken language and suppress the external noise. Considering during inference the model can not obtain ground truth video segments, we design a curriculum strategy that gradually shifts the input video from the ground truth to the entire video content during pre-training. Finally, the model can learn how to extract critical visual information from the entire video clip to help understand the spoken language. In addition, we collect the first large-scale spoken video grounding dataset based on ActivityNet, which is named as ActivityNet Speech dataset. Extensive experiments demonstrate our proposed video-guided curriculum learning can facilitate the pre-training process to obtain a mutual audio encoder, significantly promoting the performance of spoken video grounding tasks. Moreover, we prove that in the case of noisy sound, our model outperforms the method that grounding video with ASR transcripts, further demonstrating the effectiveness of our curriculum strategy.
Progressive Fusion for Multimodal Integration
Venue: arXiv cs.LG 2022
Year: 2022

Integration of multimodal information from various sources has been shown to boost the performance of machine learning models and thus has received increased attention in recent years. Often such models use deep modality-specific networks to obtain unimodal features which are combined to obtain "late-fusion" representations. However, these designs run the risk of information loss in the respective unimodal pipelines. On the other hand, "early-fusion" methodologies, which combine features early, suffer from the problems associated with feature heterogeneity and high sample complexity. In this work, we present an iterative representation refinement approach, called Progressive Fusion, which mitigates the issues with late fusion representations. Our model-agnostic technique introduces backward connections that make late stage fused representations available to early layers, improving the expressiveness of the representations at those stages, while retaining the advantages of late fusion designs. We test Progressive Fusion on tasks including affective sentiment detection, multimedia analysis, and time series fusion with different models, demonstrating its versatility. We show that our approach consistently improves performance, for instance attaining a 5% reduction in MSE and 40% improvement in robustness on multimodal time series prediction.
Memory Tagging: A Memory Efficient Design
Venue: arXiv cs.CR 2022
Year: 2022

ARM recently introduced a security feature called Memory Tagging Extension or MTE, which is designed to defend against common memory safety vulnerabilities, such as buffer overflow and use after free. In this paper, we examine three aspects of MTE. First, we survey how modern software systems, such as Glibc, Android, Chrome, Linux, and LLVM, use MTE. We identify some common weaknesses and propose improvements. Second, we develop and experiment with an architectural improvement to MTE that improves its memory efficiency. Our design enables longer memory tags, which improves the accuracy of MTE. Finally, we discuss a number of enhancements to MTE to improve its security against certain memory safety attacks.
MTS Kion Implicit Contextualised Sequential Dataset for Movie Recommendation
Venue: arXiv cs.IR 2022
Year: 2022

We present a new movie and TV show recommendation dataset collected from the real users of MTS Kion video-on-demand platform. In contrast to other popular movie recommendation datasets, such as MovieLens or Netflix, our dataset is based on the implicit interactions registered at the watching time, rather than on explicit ratings. We also provide rich contextual and side information including interactions characteristics (such as temporal information, watch duration and watch percentage), user demographics and rich movies meta-information. In addition, we describe the MTS Kion Challenge - an online recommender systems challenge that was based on this dataset - and provide an overview of the best performing solutions of the winners. We keep the competition sandbox open, so the researchers are welcome to try their own recommendation algorithms and measure the quality on the private part of the dataset.
Hidden Author Bias in Book Recommendation
Venue: arXiv cs.IR 2022
Year: 2022

Collaborative filtering algorithms have the advantage of not requiring sensitive user or item information to provide recommendations. However, they still suffer from fairness related issues, like popularity bias. In this work, we argue that popularity bias often leads to other biases that are not obvious when additional user or item information is not provided to the researcher. We examine our hypothesis in the book recommendation case on a commonly used dataset with book ratings. We enrich it with author information using publicly available external sources. We find that popular books are mainly written by US citizens in the dataset, and that these books tend to be recommended disproportionally by popular collaborative filtering algorithms compared to the users' profiles. We conclude that the societal implications of popularity bias should be further examined by the scholar community.
Intelligent Traffic Monitoring with Hybrid AI
Venue: arXiv cs.AI 2022
Year: 2022

Challenges in Intelligent Traffic Monitoring (ITMo) are exacerbated by the large quantity and modalities of data and the need for the utilization of state-of-the-art (SOTA) reasoners. We formulate the problem of ITMo and introduce HANS, a neuro-symbolic architecture for multi-modal context understanding, and its application to ITMo. HANS utilizes knowledge graph technology to serve as a backbone for SOTA reasoning in the traffic domain. Through case studies, we show how HANS addresses the challenges associated with traffic monitoring while being able to integrate with a wide range of reasoning methods
REMOT: A Region-to-Whole Framework for Realistic Human Motion Transfer
Venue: arXiv cs.CV 2022
Year: 2022

Human Video Motion Transfer (HVMT) aims to, given an image of a source person, generate his/her video that imitates the motion of the driving person. Existing methods for HVMT mainly exploit Generative Adversarial Networks (GANs) to perform the warping operation based on the flow estimated from the source person image and each driving video frame. However, these methods always generate obvious artifacts due to the dramatic differences in poses, scales, and shifts between the source person and the driving person. To overcome these challenges, this paper presents a novel REgionto-whole human MOtion Transfer (REMOT) framework based on GANs. To generate realistic motions, the REMOT adopts a progressive generation paradigm: it first generates each body part in the driving pose without flow-based warping, then composites all parts into a complete person of the driving motion. Moreover, to preserve the natural global appearance, we design a Global Alignment Module to align the scale and position of the source person with those of the driving person based on their layouts. Furthermore, we propose a Texture Alignment Module to keep each part of the person aligned according to the similarity of the texture. Finally, through extensive quantitative and qualitative experiments, our REMOT achieves state-of-the-art results on two public benchmarks.
NODE IK: Solving Inverse Kinematics with Neural Ordinary Differential Equations for Path Planning
Venue: arXiv cs.RO 2022
Year: 2022

This paper proposes a novel inverse kinematics (IK) solver of articulated robotic systems for path planning. IK is a traditional but essential problem for robot manipulation. Recently, data-driven methods have been proposed to quickly solve the IK for path planning. These methods can handle a large amount of IK requests at once with the advantage of GPUs. However, the accuracy is still low, and the model requires considerable time for training. Therefore, we propose an IK solver that improves accuracy and memory efficiency by utilizing the continuous hidden dynamics of Neural ODE. The performance is compared using multiple robots.
Predictive Semantics for Past-CTL Runtime Monitors
Venue: arXiv cs.SE 2022
Year: 2022

The distributed monitoring of swarms of devices cooperating to common global goals is becoming increasingly important, as such systems are employed for critical applications, e.g., in search and rescue missions during emergencies. In this paper, we target the distributed run-time verification of global properties of a swarm expressed as logical formulas in a temporal logic. In particular, for the implementation of decentralized monitors, we adopt the Field Calculus (FC) language, and exploit the results of previous works which have shown the possibility of automatically translating temporal logic formulas into FC programs. The main limitation of such works lies in the fact that the formulas are expressed in the past-CTL logic, which only features past modalities, and is therefore ineffective in predicting properties about the future evolution of a system. In this paper, we inject some limited prediction capability into the past-CTL logic by providing an extended semantics on a multi-valued logic, then assessing how this affects the automated translation into field calculus monitors.
Online Load Balancing on Uniform Machines with Limited Migration
Venue: arXiv cs.DS 2022
Year: 2022

In the problem of online load balancing on uniformly related machines with bounded migration, jobs arrive online one after another and have to be immediately placed on one of a given set of machines without knowledge about jobs that may arrive later on. Each job has a size and each machine has a speed, and the load due to a job assigned to a machine is obtained by dividing the first value by the second. The goal is to minimize the maximum overall load any machine receives. However, unlike in the pure online case, each time a new job arrives it contributes a migration potential equal to the product of its size and a certain migration factor. This potential can be spend to reassign jobs either right away (non-amortized case) or at any later time (amortized case). Semi-online models of this flavor have been studied intensively for several fundamental problems, e.g., load balancing on identical machines and bin packing, but uniformly related machines have not been considered up to now. In the present paper, the classical doubling strategy on uniformly related machines is combined with migration to achieve an $(8/3+\varepsilon)$-competitive algorithm and a $(4+\varepsilon)$-competitive algorithm with $O(1/\varepsilon)$ amortized and non-amortized migration, respectively, while the best known competitive ratio in the pure online setting is roughly $5.828$.
On Almost-Sure Intention Deception Planning that Exploits Imperfect Observers
Venue: arXiv cs.GT 2022
Year: 2022

Intention deception involves computing a strategy which deceives the opponent into a wrong belief about the agent's intention or objective. This paper studies a class of probabilistic planning problems with intention deception and investigates how a defender's limited sensing modality can be exploited by an attacker to achieve its attack objective almost surely (with probability one) while hiding its intention. In particular, we model the attack planning in a stochastic system modeled as a Markov decision process (MDP). The attacker is to reach some target states while avoiding unsafe states in the system and knows that his behavior is monitored by a defender with partial observations. Given partial state observations for the defender, we develop qualitative intention deception planning algorithms that construct attack strategies to play against an action-visible defender and an action-invisible defender, respectively. The synthesized attack strategy not only ensures the attack objective is satisfied almost surely but also deceives the defender into believing that the observed behavior is generated by a normal/legitimate user and thus failing to detect the presence of an attack. We show the proposed algorithms are correct and complete and illustrate the deceptive planning methods with examples.
Neural Approaches to Co-Optimization in Robotics
Venue: arXiv cs.RO 2022
Year: 2022

Robots and intelligent systems that sense or interact with the world are increasingly being used to automate a wide array of tasks. The ability of these systems to complete these tasks depends on a large range of technologies such as the mechanical and electrical parts that make up the physical body of the robot and its sensors, perception algorithms to perceive the environment, and planning and control algorithms to produce meaningful actions. Therefore, it is often necessary to consider the interactions between these components when designing an embodied system. This thesis explores work on the task-driven co-optimization of robotics systems in an end-to-end manner, simultaneously optimizing the physical components of the system with inference or control algorithms directly for task performance. We start by considering the problem of optimizing a beacon-based localization system directly for localization accuracy. Designing such a system involves placing beacons throughout the environment and inferring location from sensor readings. In our work, we develop a deep learning approach to optimize both beacon placement and location inference directly for localization accuracy. We then turn our attention to the related problem of task-driven optimization of robots and their controllers. In our work, we start by proposing a data-efficient algorithm based on multi-task reinforcement learning. Our approach efficiently optimizes both physical design and control parameters directly for task performance by leveraging a design-conditioned controller capable of generalizing over the space of physical designs. We then follow this up with an extension to allow for the optimization over discrete morphological parameters such as the number and configuration of limbs. Finally, we conclude by exploring the fabrication and deployment of optimized soft robots.
Learning with Differentiable Algorithms
Venue: arXiv cs.LG 2022
Year: 2022

Classic algorithms and machine learning systems like neural networks are both abundant in everyday life. While classic computer science algorithms are suitable for precise execution of exactly defined tasks such as finding the shortest path in a large graph, neural networks allow learning from data to predict the most likely answer in more complex tasks such as image classification, which cannot be reduced to an exact algorithm. To get the best of both worlds, this thesis explores combining both concepts leading to more robust, better performing, more interpretable, more computationally efficient, and more data efficient architectures. The thesis formalizes the idea of algorithmic supervision, which allows a neural network to learn from or in conjunction with an algorithm. When integrating an algorithm into a neural architecture, it is important that the algorithm is differentiable such that the architecture can be trained end-to-end and gradients can be propagated back through the algorithm in a meaningful way. To make algorithms differentiable, this thesis proposes a general method for continuously relaxing algorithms by perturbing variables and approximating the expectation value in closed form, i.e., without sampling. In addition, this thesis proposes differentiable algorithms, such as differentiable sorting networks, differentiable renderers, and differentiable logic gate networks. Finally, this thesis presents alternative training strategies for learning with algorithms.
Seq-UPS: Sequential Uncertainty-aware Pseudo-label Selection for Semi-Supervised Text Recognition
Venue: arXiv cs.CV 2022
Year: 2022

This paper looks at semi-supervised learning (SSL) for image-based text recognition. One of the most popular SSL approaches is pseudo-labeling (PL). PL approaches assign labels to unlabeled data before re-training the model with a combination of labeled and pseudo-labeled data. However, PL methods are severely degraded by noise and are prone to over-fitting to noisy labels, due to the inclusion of erroneous high confidence pseudo-labels generated from poorly calibrated models, thus, rendering threshold-based selection ineffective. Moreover, the combinatorial complexity of the hypothesis space and the error accumulation due to multiple incorrect autoregressive steps posit pseudo-labeling challenging for sequence models. To this end, we propose a pseudo-label generation and an uncertainty-based data selection framework for semi-supervised text recognition. We first use Beam-Search inference to yield highly probable hypotheses to assign pseudo-labels to the unlabelled examples. Then we adopt an ensemble of models, sampled by applying dropout, to obtain a robust estimate of the uncertainty associated with the prediction, considering both the character-level and word-level predictive distribution to select good quality pseudo-labels. Extensive experiments on several benchmark handwriting and scene-text datasets show that our method outperforms the baseline approaches and the previous state-of-the-art semi-supervised text-recognition methods.