OpenRaiser/Intern-Atlas huggingface dataset:
Lost in Transmission: On the Impact of Networking Corruptions on Video Machine Learning Models
Venue: arXiv cs.CV 2022
Year: 2022

We study how networking corruptions--data corruptions caused by networking errors--affect video machine learning (ML) models. We discover apparent networking corruptions in Kinetics-400, a benchmark video ML dataset. In a simulation study, we investigate (1) what artifacts networking corruptions cause, (2) how such artifacts affect ML models, and (3) whether standard robustness methods can mitigate their negative effects. We find that networking corruptions cause visual and temporal artifacts (i.e., smeared colors or frame drops). These networking corruptions degrade performance on a variety of video ML tasks, but effects vary by task and dataset, depending on how much temporal context the tasks require. Lastly, we evaluate data augmentation--a standard defense for data corruptions--but find that it does not recover performance.
A heuristic method for data allocation and task scheduling on heterogeneous multiprocessor systems under memory constraints
Venue: arXiv cs.DC 2022
Year: 2022

Computing workflows in heterogeneous multiprocessor systems are frequently modeled as directed acyclic graphs of tasks and data blocks, which represent computational modules and their dependencies in the form of data produced by a task and used by others. However, for some workflows, such as the task schedule in a digital signal processor may run out of memory by exposing too much parallelism. This paper focuses on the data allocation and task scheduling problem under memory constraints, and concentrates on shared memory platforms. We first propose an integer linear programming model to formulate the problem. Then we consider the problem as an extended flexible job shop scheduling problem, while trying to minimize the critical path of the graph. To solve this problem, we propose a tabu search algorithm (TS) which combines several distinguished features such as a greedy initial solution construction method and a mixed neighborhood evaluation strategy based on exact evaluation and approximate evaluation methods. Experimental results on randomly generated instances show that the the proposed TS algorithm can obtain relatively high-quality solutions in a reasonable computational time. In specific, the tabu search method averagely improves the makespan by 5-25\% compared to the classical load balancing algorithm that are widely used in the literature. Besides, some key features of TS are also analyzed to identify its success factors.
Vehicle-To-Pedestrian Communication Feedback Module: A Study on Increasing Legibility, Public Acceptance and Trust
Venue: arXiv cs.HC 2022
Year: 2022

Vehicle pedestrian communication is extremely important when developing autonomy for an autonomous vehicle. Enabling bidirectional nonverbal communication between pedestrians and autonomous vehicles will lead to an improvement of pedestrians' safety in autonomous driving. If a pedestrian wants to communicate, the autonomous vehicle should provide feedback to the human about what it is about to do. The user study presented in this paper investigated several possible options for an external vehicle display for effective nonverbal communication between an autonomous vehicle and a human. The result of this study will guide the development of the feedback module in future studies, optimizing for public acceptance and trust in the autonomous vehicle's decision while being legible to the widest range of potential users. The results of this study show that participants prefer symbols over text, lights and road projection. Additionally, participants prefer the combination of symbols and text as interaction modes to be displayed if the autonomous vehicle is not driving. Further, the results show that the text interaction mode option "Safe to cross" should be used combined with the symbol interaction mode option that displays a symbol of a walking person. We plan to elaborate and focus on the selected interaction modes via Virtual Reality and in the real world in ongoing and future studies.
Synthetic Over-sampling for Imbalanced Node Classification with Graph Neural Networks
Venue: arXiv cs.LG 2022
Year: 2022

In recent years, graph neural networks (GNNs) have achieved state-of-the-art performance for node classification. However, most existing GNNs would suffer from the graph imbalance problem. In many real-world scenarios, node classes are imbalanced, with some majority classes making up most parts of the graph. The message propagation mechanism in GNNs would further amplify the dominance of those majority classes, resulting in sub-optimal classification performance. In this work, we seek to address this problem by generating pseudo instances of minority classes to balance the training data, extending previous over-sampling-based techniques. This task is non-trivial, as those techniques are designed with the assumption that instances are independent. Neglection of relation information would complicate this oversampling process. Furthermore, the node classification task typically takes the semi-supervised setting with only a few labeled nodes, providing insufficient supervision for the generation of minority instances. Generated new nodes of low quality would harm the trained classifier. In this work, we address these difficulties by synthesizing new nodes in a constructed embedding space, which encodes both node attributes and topology information. Furthermore, an edge generator is trained simultaneously to model the graph structure and provide relations for new samples. To further improve the data efficiency, we also explore synthesizing mixed ``in-between'' nodes to utilize nodes from the majority class in this over-sampling process. Experiments on real-world datasets validate the effectiveness of our proposed framework.
Multi-instrument Music Synthesis with Spectrogram Diffusion
Venue: arXiv cs.SD 2022
Year: 2022

An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural synthesizers have exhibited a tradeoff between domain-specific models that offer detailed control of only specific instruments, or raw waveform models that can train on any music but with minimal control and slow generation. In this work, we focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. This enables training on a wide range of transcription datasets with a single model, which in turn offers note-level control of composition and instrumentation across a wide range of instruments. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We compare training the decoder as an autoregressive model and as a Denoising Diffusion Probabilistic Model (DDPM) and find that the DDPM approach is superior both qualitatively and as measured by audio reconstruction and Fréchet distance metrics. Given the interactivity and generality of this approach, we find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.
Downlink Power Minimization in Intelligent Reconfigurable Surface-Aided Security Classification Wireless Communications System
Venue: arXiv cs.IT 2022
Year: 2022

User privacy protection is considered a critical issue in wireless networks, which drives the demand for various secure information interaction techniques. In this paper, we introduce an intelligent reflecting surface (IRS)-aided security classification wireless communication system, which reduces the transmit power of the base station (BS) by classifying users with different security requirements. Specifically, we divide the users into confidential subscribers with secure communication requirements and general communication users with simple communication requirements. During the communication period, we guarantee the secure rate of the confidential subscribers while ensuring the service quality of the general communication users, thereby reducing the transmit power of the BS. To realize such a secure and green information transmission, the BS implements a beamforming design on the transmitted signal superimposed with artificial noise (AN) and then broadcasts it to users with the assistance of the IRS's reflection. We develop an alternating optimization framework to minimize the BS downlink power with respect to the active beamformers of the BS, the AN vector at the BS, and the reflection phase shifts of the IRS. A successive convex approximation (SCA) method is proposed so that the nonconvex beamforming problems can be converted to tractable convex forms. The simulation results demonstrate that the proposed algorithm is convergent and can reduce the transmit power by 20\% compared to the best benchmark scheme.
Multi-party Holomeetings: Toward a New Era of Low-Cost Volumetric Holographic Meetings in Virtual Reality
Venue: arXiv cs.MM 2022
Year: 2022

Fueled by advances in multi-party communications, increasingly mature immersive technologies being adopted, and the COVID-19 pandemic, a new wave of social virtual reality (VR) platforms have emerged to support socialization, interaction, and collaboration among multiple remote users who are integrated into shared virtual environments. Social VR aims to increase levels of (co-)presence and interaction quality by overcoming the limitations of 2D windowed representations in traditional multi-party video conferencing tools, although most existing solutions rely on 3D avatars to represent users. This article presents a social VR platform that supports real-time volumetric holographic representations of users that are based on point clouds captured by off-the-shelf RGB-D sensors, and it analyzes the platform's potential for conducting interactive holomeetings (i.e., holoconferencing scenarios). This work evaluates such a platform's performance and readiness for conducting meetings with up to four users, and it provides insights into aspects of the user experience when using single-camera and low-cost capture systems in scenarios with both frontal and side viewpoints. Overall, the obtained results confirm the platform's maturity and the potential of holographic communications for conducting interactive multi-party meetings, even when using low-cost systems and single-camera capture systems in scenarios where users are sitting or have a limited translational movement along the X, Y, and Z axes within the 3D virtual environment (commonly known as 3 Degrees of Freedom plus, 3DoF+)
Hierarchical Conditional Variational Autoencoder Based Acoustic Anomaly Detection
Venue: arXiv cs.LG 2022
Year: 2022

This paper aims to develop an acoustic signal-based unsupervised anomaly detection method for automatic machine monitoring. Existing approaches such as deep autoencoder (DAE), variational autoencoder (VAE), conditional variational autoencoder (CVAE) etc. have limited representation capabilities in the latent space and, hence, poor anomaly detection performance. Different models have to be trained for each different kind of machines to accurately perform the anomaly detection task. To solve this issue, we propose a new method named as hierarchical conditional variational autoencoder (HCVAE). This method utilizes available taxonomic hierarchical knowledge about industrial facility to refine the latent space representation. This knowledge helps model to improve the anomaly detection performance as well. We demonstrated the generalization capability of a single HCVAE model for different types of machines by using appropriate conditions. Additionally, to show the practicability of the proposed approach, (i) we evaluated HCVAE model on different domain and (ii) we checked the effect of partial hierarchical knowledge. Our results show that HCVAE method validates both of these points, and it outperforms the baseline system on anomaly detection task by utmost 15 % on the AUC score metric.
Model-based Offline Imitation Learning with Non-expert Data
Venue: arXiv cs.LG 2022
Year: 2022

Although Behavioral Cloning (BC) in theory suffers compounding errors, its scalability and simplicity still makes it an attractive imitation learning algorithm. In contrast, imitation approaches with adversarial training typically does not share the same problem, but necessitates interactions with the environment. Meanwhile, most imitation learning methods only utilises optimal datasets, which could be significantly more expensive to obtain than its suboptimal counterpart. A question that arises is, can we utilise the suboptimal dataset in a principled manner, which otherwise would have been idle? We propose a scalable model-based offline imitation learning algorithmic framework that leverages datasets collected by both suboptimal and optimal policies, and show that its worst case suboptimality becomes linear in the time horizon with respect to the expert samples. We empirically validate our theoretical results and show that the proposed method \textit{always} outperforms BC in the low data regime on simulated continuous control domains
Rare event failure test case generation in Learning-Enabled-Controllers
Venue: arXiv cs.LG 2022
Year: 2022

Machine learning models have prevalent applications in many real-world problems, which increases the importance of correctness in the behaviour of these trained models. Finding a good test case that can reveal the potential failure in these trained systems can help to retrain these models to increase their correctness. For a well-trained model, the occurrence of a failure is rare. Consequently, searching these rare scenarios by evaluating each sample in input search space or randomized search would be costly and sometimes intractable due to large search space, limited computational resources, and available time. In this paper, we tried to address this challenge of finding these failure scenarios faster than traditional randomized search. The central idea of our approach is to separate the input data space in region of high failure probability and region of low/minimal failure probability based on the observation made by training data, data drawn from real-world statistics, and knowledge from a domain expert. Using these information, we can design a generative model from which we can generate scenarios that have a high likelihood to reveal the potential failure. We evaluated this approach on two different experimental scenarios and able to speed up the discovery of such failures a thousand-fold faster than the traditional randomized search.
NeuGuard: Lightweight Neuron-Guided Defense against Membership Inference Attacks
Venue: arXiv cs.CR 2022
Year: 2022

Membership inference attacks (MIAs) against machine learning models can lead to serious privacy risks for the training dataset used in the model training. In this paper, we propose a novel and effective Neuron-Guided Defense method named NeuGuard against membership inference attacks (MIAs). We identify a key weakness in existing defense mechanisms against MIAs wherein they cannot simultaneously defend against two commonly used neural network based MIAs, indicating that these two attacks should be separately evaluated to assure the defense effectiveness. We propose NeuGuard, a new defense approach that jointly controls the output and inner neurons' activation with the object to guide the model output of training set and testing set to have close distributions. NeuGuard consists of class-wise variance minimization targeting restricting the final output neurons and layer-wise balanced output control aiming to constrain the inner neurons in each layer. We evaluate NeuGuard and compare it with state-of-the-art defenses against two neural network based MIAs, five strongest metric based MIAs including the newly proposed label-only MIA on three benchmark datasets. Results show that NeuGuard outperforms the state-of-the-art defenses by offering much improved utility-privacy trade-off, generality, and overhead.
Lower Bounds for Sorting 16, 17, and 18 Elements
Venue: arXiv cs.DS 2022
Year: 2022

It is a long-standing open question to determine the minimum number of comparisons $S(n)$ that suffice to sort an array of $n$ elements. Indeed, before this work $S(n)$ has been known only for $n\leq 22$ with the exception for $n=16$, $17$, and $18$. In this work, we fill that gap by proving that sorting $n=16$, $17$, and $18$ elements requires $46$, $50$, and $54$ comparisons respectively. This fully determines $S(n)$ for these values and disproves a conjecture by Knuth that $S(16) = 45$. Moreover, we show that for sorting $28$ elements at least 99 comparisons are needed. We obtain our result via an exhaustive computer search which extends previous work by Wells (1965) and Peczarski (2002, 2004, 2007, 2012). Our progress is both based on advances in hardware and on novel algorithmic ideas such as applying a bidirectional search to this problem.
Governing for Free: Rule Process Effects on Reddit Moderator Motivations
Venue: arXiv cs.CY 2022
Year: 2022

Developing a strong community requires empowered leadership capable of overcoming governance challenges. New online platforms have given users opportunities to practice governance through content moderation roles. The over 2.8 million "subreddit" communities on Reddit are governed by hundreds of thousands of volunteer moderators, many of whom have no training or prior experience in a governing role. While moderators often devote daily time to community maintenance and cope with the emotional effects of hate comments or disturbing content, Reddit provides no compensation for this position. Thus, moderators' internal motivations and desire to continue filling this role is critical for their community. Drawing upon the relationship between governance procedures and internalized motivation, we investigate how the processes through which subreddit moderators generate community rules increase moderators' motivation through the meeting of social-psychological needs: Procedural Justice and Self Determination, and Self-Other Merging. Preliminary analysis of survey data from 620 moderators across Reddit shows a correlation between moderators' administrative behaviors and the social-psychological needs underpinning their motivations. Understanding these relationships will allow us to empower moderators to build engaging and cooperative online communities.
An Unsupervised Deep-Learning Method for Bone Age Assessment
Venue: arXiv cs.CV 2022
Year: 2022

The bone age, reflecting the degree of development of the bones, can be used to predict the adult height and detect endocrine diseases of children. Both examinations of radiologists and variability of operators have a significant impact on bone age assessment. To decrease human intervention , machine learning algorithms are used to assess the bone age automatically. However, conventional supervised deep-learning methods need pre-labeled data. In this paper, based on the convolutional auto-encoder with constraints (CCAE), an unsupervised deep-learning model proposed in the classification of the fingerprint, we propose this model for the classification of the bone age and baptize it BA-CCAE. In the proposed BA-CCAE model, the key regions of the raw X-ray images of the bone age are encoded, yielding the latent vectors. The K-means clustering algorithm is used to obtain the final classifications by grouping the latent vectors of the bone images. A set of experiments on the Radiological Society of North America pediatric bone age dataset (RSNA) show that the accuracy of classifications at 48-month intervals is 76.15%. Although the accuracy now is lower than most of the existing supervised models, the proposed BA-CCAE model can establish the classification of bone age without any pre-labeled data, and to the best of our knowledge, the proposed BA-CCAE is one of the few trails using the unsupervised deep-learning method for the bone age assessment.
Security of Machine Learning-Based Anomaly Detection in Cyber Physical Systems
Venue: arXiv cs.DC 2022
Year: 2022

In this study, we focus on the impact of adversarial attacks on deep learning-based anomaly detection in CPS networks and implement a mitigation approach against the attack by retraining models using adversarial samples. We use the Bot-IoT and Modbus IoT datasets to represent the two CPS networks. We train deep learning models and generate adversarial samples using these datasets. These datasets are captured from IoT and Industrial IoT (IIoT) networks. They both provide samples of normal and attack activities. The deep learning model trained with these datasets showed high accuracy in detecting attacks. An Artificial Neural Network (ANN) is adopted with one input layer, four intermediate layers, and one output layer. The output layer has two nodes representing the binary classification results. To generate adversarial samples for the experiment, we used a function called the `fast_gradient_method' from the Cleverhans library. The experimental result demonstrates the influence of FGSM adversarial samples on the accuracy of the predictions and proves the effectiveness of using the retrained model to defend against adversarial attacks.
Bootstrapping Multi-view Representations for Fake News Detection
Venue: arXiv cs.CV 2022
Year: 2022

Previous researches on multimedia fake news detection include a series of complex feature extraction and fusion networks to gather useful information from the news. However, how cross-modal consistency relates to the fidelity of news and how features from different modalities affect the decision-making are still open questions. This paper presents a novel scheme of Bootstrapping Multi-view Representations (BMR) for fake news detection. Given a multi-modal news, we extract representations respectively from the views of the text, the image pattern and the image semantics. Improved Multi-gate Mixture-of-Expert networks (iMMoE) are proposed for feature refinement and fusion. Representations from each view are separately used to coarsely predict the fidelity of the whole news, and the multimodal representations are able to predict the cross-modal consistency. With the prediction scores, we reweigh each view of the representations and bootstrap them for fake news detection. Extensive experiments conducted on typical fake news detection datasets prove that the proposed BMR outperforms state-of-the-art schemes.
SeATrans: Learning Segmentation-Assisted diagnosis model via Transformer
Venue: arXiv cs.CV 2022
Year: 2022

Clinically, the accurate annotation of lesions/tissues can significantly facilitate the disease diagnosis. For example, the segmentation of optic disc/cup (OD/OC) on fundus image would facilitate the glaucoma diagnosis, the segmentation of skin lesions on dermoscopic images is helpful to the melanoma diagnosis, etc. With the advancement of deep learning techniques, a wide range of methods proved the lesions/tissues segmentation can also facilitate the automated disease diagnosis models. However, existing methods are limited in the sense that they can only capture static regional correlations in the images. Inspired by the global and dynamic nature of Vision Transformer, in this paper, we propose Segmentation-Assisted diagnosis Transformer (SeATrans) to transfer the segmentation knowledge to the disease diagnosis network. Specifically, we first propose an asymmetric multi-scale interaction strategy to correlate each single low-level diagnosis feature with multi-scale segmentation features. Then, an effective strategy called SeA-block is adopted to vitalize diagnosis feature via correlated segmentation features. To model the segmentation-diagnosis interaction, SeA-block first embeds the diagnosis feature based on the segmentation information via the encoder, and then transfers the embedding back to the diagnosis feature space by a decoder. Experimental results demonstrate that SeATrans surpasses a wide range of state-of-the-art (SOTA) segmentation-assisted diagnosis methods on several disease diagnosis tasks.