OpenRaiser/Intern-Atlas huggingface dataset:
Esports Agents with a Theory of Mind: Towards Better Engagement, Education, and Engineering
Venue: arXiv cs.HC 2021
Year: 2021

The role of AI in esports is shifting from leveraging games as a testbed for improving AI algorithms to addressing the needs of the esports players such as enhancing their gaming experience, esports skills, and providing coaching. For AI to be able to effectively address such needs in esports, AI agents require a theory of mind, that is, the ability to infer players' tactics and intents. To that end, in this position paper, we argue for human-in-the-loop approaches for the discovery and computational embedding of the theory of mind within behavioral models of esports players. We discuss that such approaches can be enabled by player-centric investigations on situated cognition that will expand our understanding of the cognitive and other unobservable factors that influence esports players' behaviors. We conclude by discussing the implications of such a research direction in esports as well as broader implications in engineering design and design education.
Learning to Represent and Predict Sets with Deep Neural Networks
Venue: arXiv cs.LG 2021
Year: 2021

In this thesis, we develop various techniques for working with sets in machine learning. Each input or output is not an image or a sequence, but a set: an unordered collection of multiple objects, each object described by a feature vector. Their unordered nature makes them suitable for modeling a wide variety of data, ranging from objects in images to point clouds to graphs. Deep learning has recently shown great success on other types of structured data, so we aim to build the necessary structures for sets into deep neural networks. The first focus of this thesis is the learning of better set representations (sets as input). Existing approaches have bottlenecks that prevent them from properly modeling relations between objects within the set. To address this issue, we develop a variety of techniques for different scenarios and show that alleviating the bottleneck leads to consistent improvements across many experiments. The second focus of this thesis is the prediction of sets (sets as output). Current approaches do not take the unordered nature of sets into account properly. We determine that this results in a problem that causes discontinuity issues with many set prediction tasks and prevents them from learning some extremely simple datasets. To avoid this problem, we develop two models that properly take the structure of sets into account. Various experiments show that our set prediction techniques can significantly benefit over existing approaches.
F-CAD: A Framework to Explore Hardware Accelerators for Codec Avatar Decoding
Venue: arXiv cs.AR 2021
Year: 2021

Creating virtual avatars with realistic rendering is one of the most essential and challenging tasks to provide highly immersive virtual reality (VR) experiences. It requires not only sophisticated deep neural network (DNN) based codec avatar decoders to ensure high visual quality and precise motion expression, but also efficient hardware accelerators to guarantee smooth real-time rendering using lightweight edge devices, like untethered VR headsets. Existing hardware accelerators, however, fail to deliver sufficient performance and efficiency targeting such decoders which consist of multi-branch DNNs and require demanding compute and memory resources. To address these problems, we propose an automation framework, called F-CAD (Facebook Codec avatar Accelerator Design), to explore and deliver optimized hardware accelerators for codec avatar decoding. Novel technologies include 1) a new accelerator architecture to efficiently handle multi-branch DNNs; 2) a multi-branch dynamic design space to enable fine-grained architecture configurations; and 3) an efficient architecture search for picking the optimized hardware design based on both application-specific demands and hardware resource constraints. To the best of our knowledge, F-CAD is the first automation tool that supports the whole design flow of hardware acceleration of codec avatar decoders, allowing joint optimization on decoder designs in popular machine learning frameworks and corresponding customized accelerator design with cycle-accurate evaluation. Results show that the accelerators generated by F-CAD can deliver up to 122.1 frames per second (FPS) and 91.6% hardware efficiency when running the latest codec avatar decoder. Compared to the state-of-the-art designs, F-CAD achieves 4.0X and 2.8X higher throughput, 62.5% and 21.2% higher efficiency than DNNBuilder and HybridDNN by targeting the same hardware device.
Performance Limits of Neighbor Discovery in Wireless Networks
Venue: arXiv cs.NI 2021
Year: 2021

Neighbor Discovery (ND) is the process employed by two wireless devices to discover each other. There are many different ND protocols, both in the scientific literature and also those employed in practice. All ND protocols involve devices sending beacons, and also listening for them. Protocols differ in terms of how the beacon transmissions and reception windows are scheduled, and the device sleeps in between consecutive transmissions and reception windows in order to save energy. A successful discovery constitutes a sending device's beacon overlapping with a receiving device's reception window. The goal of all ND protocols is to minimize the discovery latency. In spite of the ubiquity of ND protocols and active research on this topic for over two decades, the basic question "Given an energy budget, what is the minimum guaranteed ND latency?", however, still remains unanswered. Given the different kinds of protocols that exist, there has also been no standard way of comparing them and their performance. This paper, for the first time, answers the question on the best-achievable ND latency for a given energy budget. We derive discovery latencies for different scenarios, e.g., when both devices have the same energy budgets, and both devices have different energy budgets. We also show that some existing protocols can be parametrized such that they perform optimally. The fact that the parametrizations of some other protocols were optimal was not known before, and can now be established using our technique. Our results are restricted to the case when a few devices discover each other at a time, as is the case in most real-life scenarios. When many devices need to discover each other simultaneously, packet collisions play a dominant role in the discovery latency and how to analyze such scenarios need further study.
Subjective and Objective Quality Assessment of Mobile Gaming Video
Venue: arXiv cs.CV 2021
Year: 2021

Nowadays, with the vigorous expansion and development of gaming video streaming techniques and services, the expectation of users, especially the mobile phone users, for higher quality of experience is also growing swiftly. As most of the existing research focuses on traditional video streaming, there is a clear lack of both subjective study and objective quality models that are tailored for quality assessment of mobile gaming content. To this end, in this study, we first present a brand new Tencent Gaming Video dataset containing 1293 mobile gaming sequences encoded with three different codecs. Second, we propose an objective quality framework, namely Efficient hard-RAnk Quality Estimator (ERAQUE), that is equipped with (1) a novel hard pairwise ranking loss, which forces the model to put more emphasis on differentiating similar pairs; (2) an adapted model distillation strategy, which could be utilized to compress the proposed model efficiently without causing significant performance drop. Extensive experiments demonstrate the efficiency and robustness of our model.
Image Captioning using Multiple Transformers for Self-Attention Mechanism
Venue: arXiv cs.CV 2021
Year: 2021

Real-time image captioning, along with adequate precision, is the main challenge of this research field. The present work, Multiple Transformers for Self-Attention Mechanism (MTSM), utilizes multiple transformers to address these problems. The proposed algorithm, MTSM, acquires region proposals using a transformer detector (DETR). Consequently, MTSM achieves the self-attention mechanism by transferring these region proposals and their visual and geometrical features through another transformer and learns the objects' local and global interconnections. The qualitative and quantitative results of the proposed algorithm, MTSM, are shown on the MSCOCO dataset.
Efficient Fuzz Testing for Apache Spark Using Framework Abstraction
Venue: arXiv cs.SE 2021
Year: 2021

The emerging data-intensive applications are increasingly dependent on data-intensive scalable computing (DISC) systems, such as Apache Spark, to process large data. Despite their popularity, DISC applications are hard to test. In recent years, fuzz testing has been remarkably successful; however, it is nontrivial to apply such traditional fuzzing to big data analytics directly because: (1) the long latency of DISC systems prohibits the applicability of fuzzing, and (2) conventional branch coverage is unlikely to identify application logic from the DISC framework implementation. We devise a novel fuzz testing tool called BigFuzz that automatically generates concrete data for an input Apache Spark program. The key essence of our approach is that we abstract the dataflow behavior of the DISC framework with executable specifications and we design schema-aware mutations based on common error types in DISC applications. Our experiments show that compared to random fuzzing, BigFuzz is able to speed up the fuzzing time by 1477X, improves application code coverage by 271%, and achieves 157% improvement in detecting application errors. The demonstration video of BigFuzz is available at https://www.youtube.com/watch?v=YvYQISILQHs&feature=youtu.be.
Deep Transfer Learning for WiFi Localization
Venue: arXiv cs.LG 2021
Year: 2021

This paper studies a WiFi indoor localisation technique based on using a deep learning model and its transfer strategies. We take CSI packets collected via the WiFi standard channel sounding as the training dataset and verify the CNN model on the subsets collected in three experimental environments. We achieve a localisation accuracy of 46.55 cm in an ideal $(6.5m \times 2.5m)$ office with no obstacles, 58.30 cm in an office with obstacles, and 102.8 cm in a sports hall $(40 \times 35m)$. Then, we evaluate the transfer ability of the proposed model to different environments. The experimental results show that, for a trained localisation model, feature extraction layers can be directly transferred to other models and only the fully connected layers need to be retrained to achieve the same baseline accuracy with non-transferred base models. This can save 60% of the training parameters and reduce the training time by more than half. Finally, an ablation study of the training dataset shows that, in both office and sport hall scenarios, after reusing the feature extraction layers of the base model, only 55% of the training data is required to obtain the models' accuracy similar to the base models.
Vulnerability Detection is Just the Beginning
Venue: arXiv cs.SE 2021
Year: 2021

Vulnerability detection plays a key role in secure software development. There are many different vulnerability detection tools and techniques to choose from, and insufficient information on which vulnerability detection techniques to use and when. The goal of this research is to assist managers and other decision-makers on software projects in making informed choices about the use of different software vulnerability detection techniques through empirical analysis of the efficiency and effectiveness of each technique. We will examine the relationships between the vulnerability detection technique used to find a vulnerability, the type of vulnerability found, the exploitability of the vulnerability, and the effort needed to fix a vulnerability on two projects where we ensure all vulnerabilities found have been fixed. We will then examine how these relationships are seen in Open Source Software more broadly where practitioners may use different vulnerability detection techniques, or may not fix all vulnerabilities found due to resource constraints.
Improving Document-Level Sentiment Classification Using Importance of Sentences
Venue: arXiv cs.CL 2021
Year: 2021

Previous researchers have considered sentiment analysis as a document classification task, in which input documents are classified into predefined sentiment classes. Although there are sentences in a document that support important evidences for sentiment analysis and sentences that do not, they have treated the document as a bag of sentences. In other words, they have not considered the importance of each sentence in the document. To effectively determine polarity of a document, each sentence in the document should be dealt with different degrees of importance. To address this problem, we propose a document-level sentence classification model based on deep neural networks, in which the importance degrees of sentences in documents are automatically determined through gate mechanisms. To verify our new sentiment analysis model, we conducted experiments using the sentiment datasets in the four different domains such as movie reviews, hotel reviews, restaurant reviews, and music reviews. In the experiments, the proposed model outperformed previous state-of-the-art models that do not consider importance differences of sentences in a document. The experimental results show that the importance of sentences should be considered in a document-level sentiment classification task.
Sequential Learning on Liver Tumor Boundary Semantics and Prognostic Biomarker Mining
Venue: arXiv cs.CV 2021
Year: 2021

The boundary of tumors (hepatocellular carcinoma, or HCC) contains rich semantics: capsular invasion, visibility, smoothness, folding and protuberance, etc. Capsular invasion on tumor boundary has proven to be clinically correlated with the prognostic indicator, microvascular invasion (MVI). Investigating tumor boundary semantics has tremendous clinical values. In this paper, we propose the first and novel computational framework that disentangles the task into two components: spatial vertex localization and sequential semantic classification. (1) A HCC tumor segmentor is built for tumor mask boundary extraction, followed by polar transform representing the boundary with radius and angle. Vertex generator is used to produce fixed-length boundary vertices where vertex features are sampled on the corresponding spatial locations. (2) The sampled deep vertex features with positional embedding are mapped into a sequential space and decoded by a multilayer perceptron (MLP) for semantic classification. Extensive experiments on tumor capsule semantics demonstrate the effectiveness of our framework. Mining the correlation between the boundary semantics and MVI status proves the feasibility to integrate this boundary semantics as a valid HCC prognostic biomarker.
Virtual Reality Sickness Mitigation Methods: A Comparative Study in a Racing Game
Venue: arXiv cs.HC 2021
Year: 2021

Using virtual reality (VR) head-mounted displays (HMDs) can induce VR sickness. VR sickness can cause strong discomfort, decrease users' presence and enjoyment, especially in games, shorten the duration of the VR experience, and can even pose health risks. Previous research has explored different VR sickness mitigation methods by adding visual effects or elements. Field of View (FOV) reduction, Depth of Field (DOF) blurring, and adding a rest frame into the virtual environment are examples of such methods. Although useful in some cases, they might result in information loss. This research is the first to compare VR sickness, presence, workload to complete a search task, and information loss of these three VR sickness mitigation methods in a racing game with two levels of control. To do this, we conducted a mixed factorial user study (N = 32) with degree of control as the between-subjects factor and the VR sickness mitigation techniques as the within-subjects factor. Participants were required to find targets with three difficulty levels while steering or not steering a car in a virtual environment. Our results show that there are no significant differences in VR sickness, presence and workload among these techniques under two levels of control in our VR racing game. We also found that changing FOV dynamically or using DOF blur effects would result in information loss while adding a target reticule as a rest frame would not.
Memory-Efficient, Limb Position-Aware Hand Gesture Recognition using Hyperdimensional Computing
Venue: arXiv cs.LG 2021
Year: 2021

Electromyogram (EMG) pattern recognition can be used to classify hand gestures and movements for human-machine interface and prosthetics applications, but it often faces reliability issues resulting from limb position change. One method to address this is dual-stage classification, in which the limb position is first determined using additional sensors to select between multiple position-specific gesture classifiers. While improving performance, this also increases model complexity and memory footprint, making a dual-stage classifier difficult to implement in a wearable device with limited resources. In this paper, we present sensor fusion of accelerometer and EMG signals using a hyperdimensional computing model to emulate dual-stage classification in a memory-efficient way. We demonstrate two methods of encoding accelerometer features to act as keys for retrieval of position-specific parameters from multiple models stored in superposition. Through validation on a dataset of 13 gestures in 8 limb positions, we obtain a classification accuracy of up to 93.34%, an improvement of 17.79% over using a model trained solely on EMG. We achieve this while only marginally increasing memory footprint over a single limb position model, requiring $8\times$ less memory than a traditional dual-stage classification architecture.
Attention-driven read-aloud technology increases reading comprehension in children with reading disabilities
Venue: arXiv cs.HC 2021
Year: 2021

The paper presents the design of an assistive reading tool that integrates read-aloud technology with eye-tracking to regulate the speed of reading and support struggling readers in following the text while listening to it. The paper describes the design rationale of this approach, following the theory of auditory-visual integration, in terms of an automatic self-adaptable technique based on the reader's gaze that provides an individualized interaction experience. This tool has been assessed in a controlled experiment with 20 children (aged 8-10 years) with a diagnosis of dyslexia and a control group of 20 children with typical reading abilities. The results show that children with reading difficulties improved their comprehension scores by 24% measured on a standardized instrument for the assessment of reading comprehension, and that children with more inaccurate reading (N=9) tended to benefit more. The findings are discussed in terms of a better integration between audio and visual text information, paving the way to improve standard read-aloud technology with gaze-contingency and self-adaptable techniques to personalize the reading experience.
Trade-offs in the Design of Multimodal Interaction for Older Adults
Venue: arXiv cs.HC 2021
Year: 2021

This paper presents key aspects and trade-offs that designers and Human-Computer Interaction practitioners might encounter when designing multimodal interaction for older adults. The paper gathers literature on multimodal interaction and assistive technology, and describes a set of design challenges specific for older users. Building on these main design challenges, four trade-offs in the design of multimodal technology for this target group are presented and discussed. To highlight the relevance of the trade-offs in the design process of multimodal technology for older adults, two of the four reported trade-offs are illustrated with two user studies that explored mid-air and speech-based interaction with a tablet device. The first study investigates the design trade-offs related to redundant multimodal commands in older, middle-aged and younger adults, whereas the second one investigates the design choices related to the definition of a set of mid-air one-hand gestures and voice input commands. Further reflections highlight the design trade-offs that such considerations bring in the process, presenting an overview of the design choices involved and of their potential consequences.
Automatic Borescope Damage Assessments for Gas Turbine Blades via Deep Learning
Venue: arXiv cs.CV 2021
Year: 2021

To maximise fuel economy, bladed components in aero-engines operate close to material limits. The severe operating environment leads to in-service damage on compressor and turbine blades, having a profound and immediate impact on the performance of the engine. Current methods of blade visual inspection are mainly based on borescope imaging. During these inspections, the sentencing of components under inspection requires significant manual effort, with a lack of systematic approaches to avoid human biases. To perform fast and accurate sentencing, we propose an automatic workflow based on deep learning for detecting damage present on rotor blades using borescope videos. Building upon state-of-the-art methods from computer vision, we show that damage statistics can be presented for each blade in a blade row separately, and demonstrate the workflow on two borescope videos.
Multimodal fusion using sparse CCA for breast cancer survival prediction
Venue: arXiv cs.LG 2021
Year: 2021

Effective understanding of a disease such as cancer requires fusing multiple sources of information captured across physical scales by multimodal data. In this work, we propose a novel feature embedding module that derives from canonical correlation analyses to account for intra-modality and inter-modality correlations. Experiments on simulated and real data demonstrate how our proposed module can learn well-correlated multi-dimensional embeddings. These embeddings perform competitively on one-year survival classification of TCGA-BRCA breast cancer patients, yielding average F1 scores up to 58.69% under 5-fold cross-validation.
The MICCAI Hackathon on reproducibility, diversity, and selection of papers at the MICCAI conference
Venue: arXiv cs.CV 2021
Year: 2021

The MICCAI conference has encountered tremendous growth over the last years in terms of the size of the community, as well as the number of contributions and their technical success. With this growth, however, come new challenges for the community. Methods are more difficult to reproduce and the ever-increasing number of paper submissions to the MICCAI conference poses new questions regarding the selection process and the diversity of topics. To exchange, discuss, and find novel and creative solutions to these challenges, a new format of a hackathon was initiated as a satellite event at the MICCAI 2020 conference: The MICCAI Hackathon. The first edition of the MICCAI Hackathon covered the topics reproducibility, diversity, and selection of MICCAI papers. In the manner of a small think-tank, participants collaborated to find solutions to these challenges. In this report, we summarize the insights from the MICCAI Hackathon into immediate and long-term measures to address these challenges. The proposed measures can be seen as starting points and guidelines for discussions and actions to possibly improve the MICCAI conference with regards to reproducibility, diversity, and selection of papers.