OpenRaiser/Intern-Atlas huggingface dataset:
Temporally Consistent Video Colorization with Deep Feature Propagation and Self-regularization Learning
Venue: arXiv cs.CV 2021
Year: 2021

Video colorization is a challenging and highly ill-posed problem. Although recent years have witnessed remarkable progress in single image colorization, there is relatively less research effort on video colorization and existing methods always suffer from severe flickering artifacts (temporal inconsistency) or unsatisfying colorization performance. We address this problem from a new perspective, by jointly considering colorization and temporal consistency in a unified framework. Specifically, we propose a novel temporally consistent video colorization framework (TCVC). TCVC effectively propagates frame-level deep features in a bidirectional way to enhance the temporal consistency of colorization. Furthermore, TCVC introduces a self-regularization learning (SRL) scheme to minimize the prediction difference obtained with different time steps. SRL does not require any ground-truth color videos for training and can further improve temporal consistency. Experiments demonstrate that our method can not only obtain visually pleasing colorized video, but also achieve clearly better temporal consistency than state-of-the-art methods.
Widen The Backdoor To Let More Attackers In
Venue: arXiv cs.LG 2021
Year: 2021

As collaborative learning and the outsourcing of data collection become more common, malicious actors (or agents) which attempt to manipulate the learning process face an additional obstacle as they compete with each other. In backdoor attacks, where an adversary attempts to poison a model by introducing malicious samples into the training data, adversaries have to consider that the presence of additional backdoor attackers may hamper the success of their own backdoor. In this paper, we investigate the scenario of a multi-agent backdoor attack, where multiple non-colluding attackers craft and insert triggered samples in a shared dataset which is used by a model (a defender) to learn a task. We discover a clear backfiring phenomenon: increasing the number of attackers shrinks each attacker's attack success rate (ASR). We then exploit this phenomenon to minimize the collective ASR of attackers and maximize defender's robustness accuracy by (i) artificially augmenting the number of attackers, and (ii) indexing to remove the attacker's sub-dataset from the model for inference, hence proposing 2 defenses.
Lookup or Exploratory: What is Your Search Intent?
Venue: arXiv cs.IR 2021
Year: 2021

Search query specificity is broadly divided into two categories - Exploratory or Lookup. If a query specificity can be identified at the run time, it can be used to significantly improve the search results as well as quality of suggestions to alter the query. However, with millions of queries coming every day on a commercial search engine, it is non-trivial to develop a horizontal technique to determine query specificity at run time. Existing techniques suffer either from lack of enough training data or are dependent on information such as query length or session information. In this paper, we show that such methodologies are inadequate or at times misleading. We propose a novel methodology, to overcome these limitations. First, we demonstrate a heuristic-based method to identify Exploratory or Lookup intent queries at scale, classifying millions of queries into the two classes with a high accuracy, as shown in our experiments. Our methodology is not dependent on session data or on query length. Next, we train a transformer-based deep neural network to classify the queries into one of the two classes at run time. Our method uses a bidirectional GRU initialized with pretrained BERT-base-uncased embeddings and an augmented triplet loss to classify the intent of queries without using any session data. We also introduce a novel Semi-Greedy Iterative Training approach to fine-tune our model. Our model is deployable for real time query specificity identification with response time of less than one millisecond. Our technique is generic, and the results have valuable implications for improving the quality of search results and suggestions.
Interactive Hierarchical Guidance using Language
Venue: arXiv cs.AI 2021
Year: 2021

Reinforcement learning has been successful in many tasks ranging from robotic control, games, energy management etc. In complex real world environments with sparse rewards and long task horizons, sample efficiency is still a major challenge. Most complex tasks can be easily decomposed into high-level planning and low level control. Therefore, it is important to enable agents to leverage the hierarchical structure and decompose bigger tasks into multiple smaller sub-tasks. We introduce an approach where we use language to specify sub-tasks and a high-level planner issues language commands to a low level controller. The low-level controller executes the sub-tasks based on the language commands. Our experiments show that this method is able to solve complex long horizon planning tasks with limited human supervision. Using language has added benefit of interpretability and ability for expert humans to take over the high-level planning task and provide language commands if necessary.
Streaming on-device detection of device directed speech from voice and touch-based invocation
Venue: arXiv cs.SD 2021
Year: 2021

When interacting with smart devices such as mobile phones or wearables, the user typically invokes a virtual assistant (VA) by saying a keyword or by pressing a button on the device. However, in many cases, the VA can accidentally be invoked by the keyword-like speech or accidental button press, which may have implications on user experience and privacy. To this end, we propose an acoustic false-trigger-mitigation (FTM) approach for on-device device-directed speech detection that simultaneously handles the voice-trigger and touch-based invocation. To facilitate the model deployment on-device, we introduce a new streaming decision layer, derived using the notion of temporal convolutional networks (TCN) [1], known for their computational efficiency. To the best of our knowledge, this is the first approach that can detect device-directed speech from more than one invocation type in a streaming fashion. We compare this approach with streaming alternatives based on vanilla Average layer, and canonical LSTMs, and show: (i) that all the models show only a small degradation in accuracy compared with the invocation-specific models, and (ii) that the newly introduced streaming TCN consistently performs better or comparable with the alternatives, while mitigating device undirected speech faster in time, and with (relative) reduction in runtime peak-memory over the LSTM-based approach of 33% vs. 7%, when compared to a non-streaming counterpart.
Leveraging Experience in Lazy Search
Venue: arXiv cs.RO 2021
Year: 2021

Lazy graph search algorithms are efficient at solving motion planning problems where edge evaluation is the computational bottleneck. These algorithms work by lazily computing the shortest potentially feasible path, evaluating edges along that path, and repeating until a feasible path is found. The order in which edges are selected is critical to minimizing the total number of edge evaluations: a good edge selector chooses edges that are not only likely to be invalid, but also eliminates future paths from consideration. We wish to learn such a selector by leveraging prior experience. We formulate this problem as a Markov Decision Process (MDP) on the state of the search problem. While solving this large MDP is generally intractable, we show that we can compute oracular selectors that can solve the MDP during training. With access to such oracles, we use imitation learning to find effective policies. If new search problems are sufficiently similar to problems solved during training, the learned policy will choose a good edge evaluation ordering and solve the motion planning problem quickly. We evaluate our algorithms on a wide range of 2D and 7D problems and show that the learned selector outperforms baseline commonly used heuristics. We further provide a novel theoretical analysis of lazy search in a Bayesian framework as well as regret guarantees on our imitation learning based approach to motion planning.
Semi-Blind Multiuser Detection Under the Presence of Reconfigurable Intelligent Surfaces
Venue: arXiv cs.IT 2021
Year: 2021

A multiuser multiple-input multiple-output wireless communication system is analytically studied, which operates with the aid of a reconfigurable intelligent surface (RIS). The intermediate RIS is equipped with multiple elements and operates via random phase rotations to simultaneously serve multiple users. Independent Rayleigh fading conditions are assumed among the included channels. The system performance is analytically studied when the linear yet efficient zero-forcing detection is implemented at the receiver. In particular, the outage performance is derived in closed-form expression for different system configuration setups with regards to the available channel state information at the receiver. Further, a joint coherent/noncoherent linear detection is analytically presented. Finally, some new engineering insights are provided, such as how the channel state information and/or the volume of antenna/RIS arrays impact on the overall system performance as well as the arising efficiency on the performance/complexity tradeoff by utilizing the joint coherent/noncoherent scheme.
Digging Into Self-Supervised Learning of Feature Descriptors
Venue: arXiv cs.CV 2021
Year: 2021

Fully-supervised CNN-based approaches for learning local image descriptors have shown remarkable results in a wide range of geometric tasks. However, most of them require per-pixel ground-truth keypoint correspondence data which is difficult to acquire at scale. To address this challenge, recent weakly- and self-supervised methods can learn feature descriptors from relative camera poses or using only synthetic rigid transformations such as homographies. In this work, we focus on understanding the limitations of existing self-supervised approaches and propose a set of improvements that combined lead to powerful feature descriptors. We show that increasing the search space from in-pair to in-batch for hard negative mining brings consistent improvement. To enhance the discriminativeness of feature descriptors, we propose a coarse-to-fine method for mining local hard negatives from a wider search space by using global visual image descriptors. We demonstrate that a combination of synthetic homography transformation, color augmentation, and photorealistic image stylization produces useful representations that are viewpoint and illumination invariant. The feature descriptors learned by the proposed approach perform competitively and surpass their fully- and weakly-supervised counterparts on various geometric benchmarks such as image-based localization, sparse feature matching, and image retrieval.
Garmr: Defending the gates of PKU-based sandboxing
Venue: arXiv cs.CR 2021
Year: 2021

Memory Protection Keys for Userspace (PKU) is a recent hardware feature that allows programs to assign virtual memory pages to protection domains, and to change domain access permissions using inexpensive, unprivileged instructions. Several in-process memory isolation approaches leverage this feature to prevent untrusted code from accessing sensitive program state and data. Typically, PKU-based isolation schemes need to be used in conjunction with mitigations such as CFI because untrusted code, when compromised, can otherwise bypass the PKU access permissions using unprivileged instructions or operating system APIs. Recently, researchers proposed fully self-contained PKUbased memory isolation schemes that do not rely on other mitigations. These systems use exploit-proof call gates to transfer control between trusted and untrusted code, as well as a sandbox that prevents tampering with the PKU infrastructure from untrusted code. In this paper, we show that these solutions are not complete. We first develop two proof-of-concept attacks against a state-of-the-art PKU-based memory isolation scheme. We then present Garmr, a PKU-based sandboxing framework that can overcome limitations of existing sandboxes. We apply Garmr to several memory isolation schemes and show that it is practical, efficient and secure.
Reinforcement Learning In Two Player Zero Sum Simultaneous Action Games
Venue: arXiv cs.LG 2021
Year: 2021

Two player zero sum simultaneous action games are common in video games, financial markets, war, business competition, and many other settings. We first introduce the fundamental concepts of reinforcement learning in two player zero sum simultaneous action games and discuss the unique challenges this type of game poses. Then we introduce two novel agents that attempt to handle these challenges by using joint action Deep Q-Networks (DQN). The first agent, called the Best Response AgenT (BRAT), builds an explicit model of its opponent's policy using imitation learning, and then uses this model to find the best response to exploit the opponent's strategy. The second agent, Meta-Nash DQN, builds an implicit model of its opponent's policy in order to produce a context variable that is used as part of the Q-value calculation. An explicit minimax over Q-values is used to find actions close to Nash equilibrium. We find empirically that both agents converge to Nash equilibrium in a self-play setting for simple matrix games, while also performing well in games with larger state and action spaces. These novel algorithms are evaluated against vanilla RL algorithms as well as recent state of the art multi-agent and two agent algorithms. This work combines ideas from traditional reinforcement learning, game theory, and meta learning.
SplitPlace: Intelligent Placement of Split Neural Nets in Mobile Edge Environments
Venue: arXiv cs.DC 2021
Year: 2021

In recent years, deep learning models have become ubiquitous in industry and academia alike. Modern deep neural networks can solve one of the most complex problems today, but coming with the price of massive compute and storage requirements. This makes deploying such massive neural networks challenging in the mobile edge computing paradigm, where edge nodes are resource-constrained, hence limiting the input analysis power of such frameworks. Semantic and layer-wise splitting of neural networks for distributed processing show some hope in this direction. However, there are no intelligent algorithms that place such modular splits to edge nodes for optimal performance. This work proposes a novel placement policy, SplitPlace, for the placement of such neural network split fragments on mobile edge hosts for efficient and scalable computing.
Parallel Minimum Spanning Forest Computation using Sparse Matrix Kernels
Venue: arXiv cs.DC 2021
Year: 2021

Formulations of graph algorithms using sparse linear algebra have yielded highly scalable distributed algorithms for problems such as connectivity and shortest path computation. We develop the first formulation of the Awerbuch-Shiloach parallel minimum spanning forest (MSF) algorithm using linear algebra primitives. We introduce a multilinear kernel that operates on an adjacency matrix and two vectors. This kernel updates graph vertices by simultaneously using information from both adjacent edges and vertices. In addition, we explore optimizations to accelerate the shortcutting step in the Awerbuch-Shiloach algorithm. We implement this MSF algorithm with Cyclops, a distributed-memory library for generalized sparse tensor algebra. We analyze the parallel scalability of our implementation on the Stampede2 supercomputer.
CoRGi: Content-Rich Graph Neural Networks with Attention
Venue: arXiv cs.LG 2021
Year: 2021

Graph representations of a target domain often project it to a set of entities (nodes) and their relations (edges). However, such projections often miss important and rich information. For example, in graph representations used in missing value imputation, items - represented as nodes - may contain rich textual information. However, when processing graphs with graph neural networks (GNN), such information is either ignored or summarized into a single vector representation used to initialize the GNN. Towards addressing this, we present CoRGi, a GNN that considers the rich data within nodes in the context of their neighbors. This is achieved by endowing CoRGi's message passing with a personalized attention mechanism over the content of each node. This way, CoRGi assigns user-item-specific attention scores with respect to the words that appear in an item's content. We evaluate CoRGi on two edge-value prediction tasks and show that CoRGi is better at making edge-value predictions over existing methods, especially on sparse regions of the graph.
Human Factors Considerations in Satellite Operation's Human-Computer Interaction Technologies: A Review of Current Applications and Theory
Venue: arXiv cs.HC 2021
Year: 2021

Satellite operations are a subset of remote operations that draw similarities with remotely piloted aircraft (RPA) and uncrewed aerial vehicle (UAV) operations. Increased research into boredom, complacency, habituation, and vigilance as they relate to satellite operations is required due to a lack of prevalence in the literature. Circadian rhythms, crew resource management, and shift work dynamics may exacerbate complacency-driven automation bias and social loafing errors in satellite operations. This overview of theory and applications aims to specifically focus on satellite operations literature within human factors research to identify areas requiring an expansion of knowledge. The human-in-the-loop commonality enables human factors lessons to be passed to satellite operations from unrelated sectors to mitigate catastrophic human error potentially. As such, this literature review details the need for increased research in satellite operations human factors.
Influencing the Influencers: Evaluating Person-to-Person Influence on Social Networks Using Granger Causality
Venue: arXiv cs.SI 2021
Year: 2021

We introduce a novel method for analyzing person-to-person content influence on Twitter. Using an Ego-Alter framework and Granger Causality, we examine President Donald Trump (the Ego) and the people he retweets (Alters) as a case study. We find that each Alter has a different scope of influence across multiple topics, different magnitude of influence on a given topic, and the magnitude of a single Alter's influence can vary across topics. This work is novel in its focus on person-to-person influence and content-based influence. Its impact is two-fold: (1) identifying "canaries in the coal mine" who could be observed by misinformation researchers or platforms to identify misinformation narratives before super-influencers spread them to large audiences, and (2) enabling digital marketing targeted toward upstream Alters of super-influencers.
Kernel Learning For Sound Field Estimation With L1 and L2 Regularizations
Venue: arXiv cs.SD 2021
Year: 2021

A method to estimate an acoustic field from discrete microphone measurements is proposed. A kernel-interpolation-based method using the kernel function formulated for sound field interpolation has been used in various applications. The kernel function with directional weighting makes it possible to incorporate prior information on source directions to improve estimation accuracy. However, in prior studies, parameters for directional weighting have been empirically determined. We propose a method to optimize these parameters using observation values, which is particularly useful when prior information on source directions is uncertain. The proposed algorithm is based on discretization of the parameters and representation of the kernel function as a weighted sum of sub-kernels. Two types of regularization for the weights, $L_1$ and $L_2$, are investigated. Experimental results indicate that the proposed method achieves higher estimation accuracy than the method without kernel learning.
Blockchain for Edge of Things: Applications, Opportunities, and Challenges
Venue: arXiv cs.CR 2021
Year: 2021

In recent years, blockchain networks have attracted significant attention in many research areas beyond cryptocurrency, one of them being the Edge of Things (EoT) that is enabled by the combination of edge computing and the Internet of Things (IoT). In this context, blockchain networks enabled with unique features such as decentralization, immutability, and traceability, have the potential to reshape and transform the conventional EoT systems with higher security levels. Particularly, the convergence of blockchain and EoT leads to a new paradigm, called BEoT that has been regarded as a promising enabler for future services and applications. In this paper, we present a state-of-the-art review of recent developments in BEoT technology and discover its great opportunities in many application domains. We start our survey by providing an updated introduction to blockchain and EoT along with their recent advances. Subsequently, we discuss the use of BEoT in a wide range of industrial applications, from smart transportation, smart city, smart healthcare to smart home and smart grid. Security challenges in BEoT paradigm are also discussed and analyzed, with some key services such as access authentication, data privacy preservation, attack detection, and trust management. Finally, some key research challenges and future directions are also highlighted to instigate further research in this promising area.
Bid Optimization using Maximum Entropy Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

Real-time bidding (RTB) has become a critical way of online advertising. In RTB, an advertiser can participate in bidding ad impressions to display its advertisements. The advertiser determines every impression's bidding price according to its bidding strategy. Therefore, a good bidding strategy can help advertisers improve cost efficiency. This paper focuses on optimizing a single advertiser's bidding strategy using reinforcement learning (RL) in RTB. Unfortunately, it is challenging to optimize the bidding strategy through RL at the granularity of impression due to the highly dynamic nature of the RTB environment. In this paper, we first utilize a widely accepted linear bidding function to compute every impression's base price and optimize it by a mutable adjustment factor derived from the RTB auction environment, to avoid optimizing every impression's bidding price directly. Specifically, we use the maximum entropy RL algorithm (Soft Actor-Critic) to optimize the adjustment factor generation policy at the impression-grained level. Finally, the empirical study on a public dataset demonstrates that the proposed bidding strategy has superior performance compared with the baselines.