OpenRaiser/Intern-Atlas huggingface dataset:
Future Frame Prediction for Robot-assisted Surgery
Venue: arXiv cs.CV 2021
Year: 2021

Predicting future frames for robotic surgical video is an interesting, important yet extremely challenging problem, given that the operative tasks may have complex dynamics. Existing approaches on future prediction of natural videos were based on either deterministic models or stochastic models, including deep recurrent neural networks, optical flow, and latent space modeling. However, the potential in predicting meaningful movements of robots with dual arms in surgical scenarios has not been tapped so far, which is typically more challenging than forecasting independent motions of one arm robots in natural scenarios. In this paper, we propose a ternary prior guided variational autoencoder (TPG-VAE) model for future frame prediction in robotic surgical video sequences. Besides content distribution, our model learns motion distribution, which is novel to handle the small movements of surgical tools. Furthermore, we add the invariant prior information from the gesture class into the generation process to constrain the latent space of our model. To our best knowledge, this is the first time that the future frames of dual arm robots are predicted considering their unique characteristics relative to general robotic videos. Experiments demonstrate that our model gains more stable and realistic future frame prediction scenes with the suturing task on the public JIGSAWS dataset.
Impact of Task Cycle Pattern on Project Success in Software Crowdsourcing
Venue: arXiv cs.SE 2021
Year: 2021

Crowdsourcing is becoming an accepted method of software development for different phases in the production lifecycle. Ideally, mass parallel production through Crowdsourcing could be an option for rapid acquisition in software engineering by leveraging infinite worker resource on the internet. It is important to understand the patterns and strategies of decomposing and uploading parallel tasks to maintain a stable worker supply as well as a satisfactory task completion rate. This research report is an empirical analysis of the available tasks' lifecycle patterns in crowdsourcing. Following the waterfall model in Crowdsourced Software Development (CSD), this research identified four patterns for the sequence of task arrival per project: 1) Prior Cycle, 2) Current Cycle, 3) Orbit Cycle, and 4) Fresh Cycle.
Fast Consensus via the Unconstrained Undecided State Dynamics
Venue: arXiv cs.DC 2021
Year: 2021

We consider the plurality consensus problem among $n$ agents. Initially, each agent has one of $k$ different opinions. Agents choose random interaction partners and revise their state according to a fixed transition function, depending on their own state and the state of the interaction partners. The goal is to reach a consensus configuration in which all agents agree on the same opinion, and if there is initially a sufficiently large bias towards one opinion, that opinion should prevail. We analyze a synchronized variant of the undecided state dynamics defined as follows. The agents act in phases, consisting of a decision and a boosting part. In the decision part, any agent that encounters an agent with a different opinion becomes undecided. In the boosting part, undecided agents adopt the first opinion they encounter. We consider this dynamics in the population model and the gossip model. For the population model, our protocol reaches consensus (w.h.p.) in $O(\log^2 n)$ parallel time, providing the first polylogarithmic result for $k > 2$ (w.h.p.) in this model. Without any assumption on the bias, fast consensus has only been shown for $k = 2$ for the unsynchronized version of the undecided state dynamics [Clementi et al., MFCS'18]. We show that the synchronized variant of the undecided state dynamics reaches consensus (w.h.p.) in time $O(\log^2 n)$, independently of the initial number, bias, or distribution of opinions. In both models, we guarantee that if there is an initial bias of $Ω(\sqrt{n \log n})$, then (w.h.p.) that opinion wins. A simple extension of our protocol in the gossip model yields a dynamics that does not depend on $n$ or $k$, is anonymous, and has (w.h.p.) runtime $O(\log^2 n)$. This solves an open problem formulated by Becchetti et al.~[Distributed Computing,~2017].
Neural Multi-Hop Reasoning With Logical Rules on Biomedical Knowledge Graphs
Venue: arXiv cs.LG 2021
Year: 2021

Biomedical knowledge graphs permit an integrative computational approach to reasoning about biological systems. The nature of biological data leads to a graph structure that differs from those typically encountered in benchmarking datasets. To understand the implications this may have on the performance of reasoning algorithms, we conduct an empirical study based on the real-world task of drug repurposing. We formulate this task as a link prediction problem where both compounds and diseases correspond to entities in a knowledge graph. To overcome apparent weaknesses of existing algorithms, we propose a new method, PoLo, that combines policy-guided walks based on reinforcement learning with logical rules. These rules are integrated into the algorithm by using a novel reward function. We apply our method to Hetionet, which integrates biomedical information from 29 prominent bioinformatics databases. Our experiments show that our approach outperforms several state-of-the-art methods for link prediction while providing interpretability.
MSMatch: Semi-Supervised Multispectral Scene Classification with Few Labels
Venue: arXiv cs.LG 2021
Year: 2021

Supervised learning techniques are at the center of many tasks in remote sensing. Unfortunately, these methods, especially recent deep learning methods, often require large amounts of labeled data for training. Even though satellites acquire large amounts of data, labeling the data is often tedious, expensive and requires expert knowledge. Hence, improved methods that require fewer labeled samples are needed. We present MSMatch, the first semi-supervised learning approach competitive with supervised methods on scene classification on the EuroSAT and UC Merced Land Use benchmark datasets. We test both RGB and multispectral images of EuroSAT and perform various ablation studies to identify the critical parts of the model. The trained neural network achieves state-of-the-art results on EuroSAT with an accuracy that is up to 19.76% better than previous methods depending on the number of labeled training examples. With just five labeled examples per class, we reach 94.53% and 95.86% accuracy on the EuroSAT RGB and multispectral datasets, respectively. On the UC Merced Land Use dataset, we outperform previous works by up to 5.59% and reach 90.71% with five labeled examples. Our results show that MSMatch is capable of greatly reducing the requirements for labeled data. It translates well to multispectral data and should enable various applications that are currently infeasible due to a lack of labeled data. We provide the source code of MSMatch online to enable easy reproduction and quick adoption.
Improved Approximation Algorithms for 2-Dimensional Knapsack: Packing into Multiple L-Shapes, Spirals, and More
Venue: arXiv cs.CG 2021
Year: 2021

In the \textsc{2-Dimensional Knapsack} problem (2DK) we are given a square knapsack and a collection of $n$ rectangular items with integer sizes and profits. Our goal is to find the most profitable subset of items that can be packed non-overlappingly into the knapsack. The currently best known polynomial-time approximation factor for 2DK is $17/9+\varepsilon<1.89$ and there is a $(3/2+\varepsilon)$-approximation algorithm if we are allowed to rotate items by 90 degrees~{[}Gálvez et al., FOCS 2017{]}. In this paper, we give $(4/3+\varepsilon)$-approximation algorithms in polynomial time for both cases, assuming that all input data are {integers polynomially bounded in $n$}. Gálvez et al.'s algorithm for 2DK partitions the knapsack into a constant number of rectangular regions plus \emph{one} L-shaped region and packs items into those {in a structured way}. We generalize this approach by allowing up to a \emph{constant} number of {\emph{more general}} regions that can have the shape of an L, a U, a Z, a spiral, and more, and therefore obtain an improved approximation ratio. {In particular, we present an algorithm that computes the essentially optimal structured packing into these regions. }
Localization of Cochlear Implant Electrodes from Cone Beam Computed Tomography using Particle Belief Propagation
Venue: arXiv cs.CV 2021
Year: 2021

Cochlear implants (CIs) are implantable medical devices that can restore the hearing sense of people suffering from profound hearing loss. The CI uses a set of electrode contacts placed inside the cochlea to stimulate the auditory nerve with current pulses. The exact location of these electrodes may be an important parameter to improve and predict the performance with these devices. Currently the methods used in clinics to characterize the geometry of the cochlea as well as to estimate the electrode positions are manual, error-prone and time consuming. We propose a Markov random field (MRF) model for CI electrode localization for cone beam computed tomography (CBCT) data-sets. Intensity and shape of electrodes are included as prior knowledge as well as distance and angles between contacts. MRF inference is based on slice sampling particle belief propagation and guided by several heuristics. A stochastic search finds the best maximum a posteriori estimation among sampled MRF realizations. We evaluate our algorithm on synthetic and real CBCT data-sets and compare its performance with two state of the art algorithms. An increase of localization precision up to 31.5% (mean), or 48.6% (median) respectively, on real CBCT data-sets is shown.
Please Don't Go -- Increasing Women's Participation in Open Source Software
Venue: arXiv cs.SE 2021
Year: 2021

Women represent less than 24% of the software development industry and suffer from various types of prejudice and biases. In Open Source Software projects, despite a variety of efforts to increase diversity and multi-gendered participation, women are even more underrepresented (less than 10%). My research focuses on answering the question: How can OSS communities increase women's participation in OSS projects? I will identify the different OSS career pathways, and develop a holistic view of women's motivations to join or leave OSS, along with their definitions of success. Based on this empirical investigation, I will work together with the Linux Foundation to design attraction and retention strategies focused on women. Before and after implementing the strategies, I will conduct empirical studies to evaluate the state of the practice and understand the implications of the strategies.
Neural Networks for Semantic Gaze Analysis in XR Settings
Venue: arXiv cs.CV 2021
Year: 2021

Virtual-reality (VR) and augmented-reality (AR) technology is increasingly combined with eye-tracking. This combination broadens both fields and opens up new areas of application, in which visual perception and related cognitive processes can be studied in interactive but still well controlled settings. However, performing a semantic gaze analysis of eye-tracking data from interactive three-dimensional scenes is a resource-intense task, which so far has been an obstacle to economic use. In this paper we present a novel approach which minimizes time and information necessary to annotate volumes of interest (VOIs) by using techniques from object recognition. To do so, we train convolutional neural networks (CNNs) on synthetic data sets derived from virtual models using image augmentation techniques. We evaluate our method in real and virtual environments, showing that the method can compete with state-of-the-art approaches, while not relying on additional markers or preexisting databases but instead offering cross-platform use.
Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons
Venue: arXiv cs.LG 2021
Year: 2021

With Open AI's publishing of their CLIP model (Contrastive Language-Image Pre-training), multi-modal neural networks now provide accessible models that combine reading with visual recognition. Their network offers novel ways to probe its dual abilities to read text while classifying visual objects. This paper demonstrates several new categories of adversarial attacks, spanning basic typographical, conceptual, and iconographic inputs generated to fool the model into making false or absurd classifications. We demonstrate that contradictory text and image signals can confuse the model into choosing false (visual) options. Like previous authors, we show by example that the CLIP model tends to read first, look later, a phenomenon we describe as reading isn't believing.
Super-convergence and Differential Privacy: Training faster with better privacy guarantees
Venue: arXiv cs.LG 2021
Year: 2021

The combination of deep neural networks and Differential Privacy has been of increasing interest in recent years, as it offers important data protection guarantees to the individuals of the training datasets used. However, using Differential Privacy in the training of neural networks comes with a set of shortcomings, like a decrease in validation accuracy and a significant increase in the use of resources and time in training. In this paper, we examine super-convergence as a way of greatly increasing training speed of differentially private neural networks, addressing the shortcoming of high training time and resource use. Super-convergence allows for acceleration in network training using very high learning rates, and has been shown to achieve models with high utility in orders of magnitude less training iterations than conventional ways. Experiments in this paper show that this order-of-magnitude speedup can also be seen when combining it with Differential Privacy, allowing for higher validation accuracies in much fewer training iterations compared to non-private, non-super convergent baseline models. Furthermore, super-convergence is shown to improve the privacy guarantees of private models.
Resilient Cooperative Adaptive Cruise Control for Autonomous Vehicles Using Machine Learning
Venue: arXiv cs.CR 2021
Year: 2021

Cooperative Adaptive Cruise Control (CACC) is a fundamental connected vehicle application that extends Adaptive Cruise Control by exploiting vehicle-to-vehicle (V2V) communication. CACC is a crucial ingredient for numerous autonomous vehicle functionalities including platooning, distributed route management, etc. Unfortunately, malicious V2V communications can subvert CACC, leading to string instability and road accidents. In this paper, we develop a novel resiliency infrastructure, RACCON, for detecting and mitigating V2V attacks on CACC. RACCON uses machine learning to develop an on-board prediction model that captures anomalous vehicular responses and performs mitigation in real time. RACCON-enabled vehicles can exploit the high efficiency of CACC without compromising safety, even under potentially adversarial scenarios. We present extensive experimental evaluation to demonstrate the efficacy of RACCON.
KGTorrent: A Dataset of Python Jupyter Notebooks from Kaggle
Venue: arXiv cs.DB 2021
Year: 2021

Computational notebooks have become the tool of choice for many data scientists and practitioners for performing analyses and disseminating results. Despite their increasing popularity, the research community cannot yet count on a large, curated dataset of computational notebooks. In this paper, we fill this gap by introducing KGTorrent, a dataset of Python Jupyter notebooks with rich metadata retrieved from Kaggle, a platform hosting data science competitions for learners and practitioners with any levels of expertise. We describe how we built KGTorrent, and provide instructions on how to use it and refresh the collection to keep it up to date. Our vision is that the research community will use KGTorrent to study how data scientists, especially practitioners, use Jupyter Notebook in the wild and identify potential shortcomings to inform the design of its future extensions.
DCF-ASN: Coarse-to-fine Real-time Visual Tracking via Discriminative Correlation Filter and Attentional Siamese Network
Venue: arXiv cs.CV 2021
Year: 2021

Discriminative correlation filters (DCF) and siamese networks have achieved promising performance on visual tracking tasks thanks to their superior computational efficiency and reliable similarity metric learning, respectively. However, how to effectively take advantages of powerful deep networks, while maintaining the real-time response of DCF, remains a challenging problem. Embedding the cross-correlation operator as a separate layer into siamese networks is a popular choice to enhance the tracking accuracy. Being a key component of such a network, the correlation layer is updated online together with other parts of the network. Yet, when facing serious disturbance, fused trackers may still drift away from the target completely due to accumulated errors. To address these issues, we propose a coarse-to-fine tracking framework, which roughly infers the target state via an online-updating DCF module first and subsequently, finely locates the target through an offline-training asymmetric siamese network (ASN). Benefitting from the guidance of DCF and the learned channel weights obtained through exploiting the given ground-truth template, ASN refines feature representation and implements precise target localization. Systematic experiments on five popular tracking datasets demonstrate that the proposed DCF-ASN achieves the state-of-the-art performance while exhibiting good tracking efficiency.
Training image classifiers using Semi-Weak Label Data
Venue: arXiv cs.LG 2021
Year: 2021

In Multiple Instance learning (MIL), weak labels are provided at the bag level with only presence/absence information known. However, there is a considerable gap in performance in comparison to a fully supervised model, limiting the practical applicability of MIL approaches. Thus, this paper introduces a novel semi-weak label learning paradigm as a middle ground to mitigate the problem. We define semi-weak label data as data where we know the presence or absence of a given class and the exact count of each class as opposed to knowing the label proportions. We then propose a two-stage framework to address the problem of learning from semi-weak labels. It leverages the fact that counting information is non-negative and discrete. Experiments are conducted on generated samples from CIFAR-10. We compare our model with a fully-supervised setting baseline, a weakly-supervised setting baseline and learning from pro-portion (LLP) baseline. Our framework not only outperforms both baseline models for MIL-based weakly super-vised setting and learning from proportion setting, but also gives comparable results compared to the fully supervised model. Further, we conduct thorough ablation studies to analyze across datasets and variation with batch size, losses architectural changes, bag size and regularization
Hyperspectral Image Super-Resolution in Arbitrary Input-Output Band Settings
Venue: arXiv cs.CV 2021
Year: 2021

Hyperspectral image (HSI) with narrow spectral bands can capture rich spectral information, but it sacrifices its spatial resolution in the process. Many machine-learning-based HSI super-resolution (SR) algorithms have been proposed recently. However, one of the fundamental limitations of these approaches is that they are highly dependent on image and camera settings and can only learn to map an input HSI with one specific setting to an output HSI with another. However, different cameras capture images with different spectral response functions and bands numbers due to the diversity of HSI cameras. Consequently, the existing machine-learning-based approaches fail to learn to super-resolve HSIs for a wide variety of input-output band settings. We propose a single Meta-Learning-Based Super-Resolution (MLSR) model, which can take in HSI images at an arbitrary number of input bands' peak wavelengths and generate SR HSIs with an arbitrary number of output bands' peak wavelengths. We leverage NTIRE2020 and ICVL datasets to train and validate the performance of the MLSR model. The results show that the single proposed model can successfully generate super-resolved HSI bands at arbitrary input-output band settings. The results are better or at least comparable to baselines that are separately trained on a specific input-output band setting.
Low differentially uniform permutations from Dobbertin APN function over $\mathbb{F}_{2^n}$
Venue: arXiv cs.CR 2021
Year: 2021

Block ciphers use S-boxes to create confusion in the cryptosystems. Such S-boxes are functions over $\mathbb{F}_{2^{n}}$. These functions should have low differential uniformity, high nonlinearity, and high algebraic degree in order to resist differential attacks, linear attacks, and higher order differential attacks, respectively. In this paper, we construct new classes of differentially $4$ and $6$-uniform permutations by modifying the image of the Dobbertin APN function $x^{d}$ with $d=2^{4k}+2^{3k}+2^{2k}+2^{k}-1$ over a subfield of $\mathbb{F}_{2^{n}}$. Furthermore, the algebraic degree and the lower bound of the nonlinearity of the constructed functions are given.
Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
Venue: arXiv cs.RO 2021
Year: 2021

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this paper proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely expert demonstration, policy derivation, and reinforcement learning. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the reinforcement learning step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60\% improvement compared to soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert.