OpenRaiser/Intern-Atlas huggingface dataset:
FashionSearchNet-v2: Learning Attribute Representations with Localization for Image Retrieval with Attribute Manipulation
Venue: arXiv cs.CV 2021
Year: 2021

The focus of this paper is on the problem of image retrieval with attribute manipulation. Our proposed work is able to manipulate the desired attributes of the query image while maintaining its other attributes. For example, the collar attribute of the query image can be changed from round to v-neck to retrieve similar images from a large dataset. A key challenge in e-commerce is that images have multiple attributes where users would like to manipulate and it is important to estimate discriminative feature representations for each of these attributes. The proposed FashionSearchNet-v2 architecture is able to learn attribute specific representations by leveraging on its weakly-supervised localization module, which ignores the unrelated features of attributes in the feature space, thus improving the similarity learning. The network is jointly trained with the combination of attribute classification and triplet ranking loss to estimate local representations. These local representations are then merged into a single global representation based on the instructed attribute manipulation where desired images can be retrieved with a distance metric. The proposed method also provides explainability for its retrieval process to help provide additional information on the attention of the network. Experiments performed on several datasets that are rich in terms of the number of attributes show that FashionSearchNet-v2 outperforms the other state-of-the-art attribute manipulation techniques. Different than our earlier work (FashionSearchNet), we propose several improvements in the learning procedure and show that the proposed FashionSearchNet-v2 can be generalized to different domains other than fashion.
How Deep Are the Fakes? Focusing on Audio Deepfake: A Survey
Venue: arXiv cs.SD 2021
Year: 2021

Deepfake is content or material that is synthetically generated or manipulated using artificial intelligence (AI) methods, to be passed off as real and can include audio, video, image, and text synthesis. This survey has been conducted with a different perspective compared to existing survey papers, that mostly focus on just video and image deepfakes. This survey not only evaluates generation and detection methods in the different deepfake categories, but mainly focuses on audio deepfakes that are overlooked in most of the existing surveys. This paper critically analyzes and provides a unique source of audio deepfake research, mostly ranging from 2016 to 2020. To the best of our knowledge, this is the first survey focusing on audio deepfakes in English. This survey provides readers with a summary of 1) different deepfake categories 2) how they could be created and detected 3) the most recent trends in this domain and shortcomings in detection methods 4) audio deepfakes, how they are created and detected in more detail which is the main focus of this paper. We found that Generative Adversarial Networks(GAN), Convolutional Neural Networks (CNN), and Deep Neural Networks (DNN) are common ways of creating and detecting deepfakes. In our evaluation of over 140 methods we found that the majority of the focus is on video deepfakes and in particular in the generation of video deepfakes. We found that for text deepfakes there are more generation methods but very few robust methods for detection, including fake news detection, which has become a controversial area of research because of the potential of heavy overlaps with human generation of fake content. This paper is an abbreviated version of the full survey and reveals a clear need to research audio deepfakes and particularly detection of audio deepfakes.
Search for Optimal Systolic Arrays: A Comprehensive Automated Exploration Framework and Lessons Learned
Venue: arXiv cs.AR 2021
Year: 2021

Systolic arrays have been widely used for accelerating HPC and deep learning applications. There is a plethora of previous works on the performance tuning of systolic arrays, but usually based on a number of oversimplified assumptions (e.g., only considering divisors for loop tiling, pruning based on off-chip data communication) to reduce the design space. In this paper, we present a comprehensive design space exploration tool named Odyssey for systolic array optimization. Odyssey does not rely on artificial assumptions to limit the design space, and yet it is highly efficient and scalable with a hybrid optimization technique. For example, for a 1024x1024x1024 matrix multiplication, it finds designs that reach 90% of the optimal performance in 5 seconds with a single CPU thread. Moreover, using Odyssey, we unveil and quantify the suboptimality introduced by multiple commonly used oversimplifications in prior studies for systolic array design space exploration. For example, Odyssey results show that limiting to divisors for loop tiling leads to a 39% performance loss, and pruning based on off-chip data movement results in a 45% performance loss. We applied Odyssey to explore the architecture trade-offs for matrix multiplication and convolutional neural network, providing inspiration into possible optimizations for these two applications.
Towards Autonomous Driving of Personal Mobility with Small and Noisy Dataset using Tsallis-statistics-based Behavioral Cloning
Venue: arXiv cs.LG 2021
Year: 2021

Autonomous driving has made great progress and been introduced in practical use step by step. On the other hand, the concept of personal mobility is also getting popular, and its autonomous driving specialized for individual drivers is expected for a new step. However, it is difficult to collect a large driving dataset, which is basically required for the learning of autonomous driving, from the individual driver of the personal mobility. In addition, when the driver is not familiar with the operation of the personal mobility, the dataset will contain non-optimal data. This study therefore focuses on an autonomous driving method for the personal mobility with such a small and noisy, so-called personal, dataset. Specifically, we introduce a new loss function based on Tsallis statistics that weights gradients depending on the original loss function and allows us to exclude noisy data in the optimization phase. In addition, we improve the visualization technique to verify whether the driver and the controller have the same region of interest. From the experimental results, we found that the conventional autonomous driving failed to drive properly due to the wrong operations in the personal dataset, and the region of interest was different from that of the driver. In contrast, the proposed method learned robustly against the errors and successfully drove automatically while paying attention to the similar region to the driver. Attached video is also uploaded on youtube: https://youtu.be/KEq8-bOxYQA
Proceedings of the CSCW 2021 Workshop -- Investigating and Mitigating Biases in Crowdsourced Data
Venue: arXiv cs.HC 2021
Year: 2021

This volume contains the position papers presented at CSCW 2021 Workshop - Investigating and Mitigating Biases in Crowdsourced Data, held online on 23rd October 2021, at the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2021). The workshop explored how specific crowdsourcing workflows, worker attributes, and work practices contribute to biases in data. The workshop also included discussions on research directions to mitigate labelling biases, particularly in a crowdsourced context, and the implications of such methods for the workers.
Attention-based Feature Decomposition-Reconstruction Network for Scene Text Detection
Venue: arXiv cs.CV 2021
Year: 2021

Recently, scene text detection has been a challenging task. Texts with arbitrary shape or large aspect ratio are usually hard to detect. Previous segmentation-based methods can describe curve text more accurately but suffer from over segmentation and text adhesion. In this paper, we propose attention-based feature decomposition-reconstruction network for scene text detection, which utilizes contextual information and low-level feature to enhance the performance of segmentation-based text detector. In the phase of feature fusion, we introduce cross level attention module to enrich contextual information of text by adding attention mechanism on fused multi-scaled feature. In the phase of probability map generation, a feature decomposition-reconstruction module is proposed to alleviate the over segmentation problem of large aspect ratio text, which decomposes text feature according to their frequency characteristic and then reconstructs it by adding low-level feature. Experiments have been conducted on two public benchmark datasets and results show that our proposed method achieves state-of-the-art performance.
Mixed Precision DNN Qunatization for Overlapped Speech Separation and Recognition
Venue: arXiv cs.SD 2021
Year: 2021

Recognition of overlapped speech has been a highly challenging task to date. State-of-the-art multi-channel speech separation system are becoming increasingly complex and expensive for practical applications. To this end, low-bit neural network quantization provides a powerful solution to dramatically reduce their model size. However, current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different model components to quantization errors. In this paper, novel mixed precision DNN quantization methods are proposed by applying locally variable bit-widths to individual TCN components of a TF masking based multi-channel speech separation system. The optimal local precision settings are automatically learned using three techniques. The first two approaches utilize quantization sensitivity metrics based on either the mean square error (MSE) loss function curvature, or the KL-divergence measured between full precision and quantized separation models. The third approach is based on mixed precision neural architecture search. Experiments conducted on the LRS3-TED corpus simulated overlapped speech data suggest that the proposed mixed precision quantization techniques consistently outperform the uniform precision baseline speech separation systems of comparable bit-widths in terms of SI-SNR and PESQ scores as well as word error rate (WER) reductions up to 2.88% absolute (8% relative).
User Allocation in Heterogeneous Network Supplied by Renewable Energy Sources
Venue: arXiv cs.NI 2021
Year: 2021

The information and communication technology, ICT, a domain of the global economy, consumes more and more energy, what is the effect of continuously increasing wireless data traffic. Thus, there is a need for more energy-efficient and sustainable network and resource management, and such a goal can be achieved by the utilization of renewable energy sources. This paper proposes a novel user allocation scheme applicable to the heterogeneous radio access networks supplied with renewable energy sources. Presented results proved that high-energy efficiency can be accomplished by reallocating some traffic to the green wireless base station.
Local planar domination revisited
Venue: arXiv cs.DC 2021
Year: 2021

We show how to compute a 20-approximation of a minimum dominating set in a planar graph in a constant number of rounds in the LOCAL model of distributed computing. This improves on the previously best known approximation factor of 52, which was achieved by an elegant and simple algorithm of Lenzen et al. Our algorithm combines ideas from the algorithm of Lenzen et al. with recent work of Czygrinow et al. and Kublenz et al. to reduce to the case of bounded degree graphs, where we can simulate a distributed version of the classical greedy algorithm.
Conceptually Diverse Base Model Selection for Meta-Learners in Concept Drifting Data Streams
Venue: arXiv cs.LG 2021
Year: 2021

Meta-learners and ensembles aim to combine a set of relevant yet diverse base models to improve predictive performance. However, determining an appropriate set of base models is challenging, especially in online environments where the underlying distribution of data can change over time. In this paper, we present a novel approach for estimating the conceptual similarity of base models, which is calculated using the Principal Angles (PAs) between their underlying subspaces. We propose two methods that use conceptual similarity as a metric to obtain a relevant yet diverse subset of base models: (i) parameterised threshold culling and (ii) parameterless conceptual clustering. We evaluate these methods against thresholding using common ensemble pruning metrics, namely predictive performance and Mutual Information (MI), in the context of online Transfer Learning (TL), using both synthetic and real-world data. Our results show that conceptual similarity thresholding has a reduced computational overhead, and yet yields comparable predictive performance to thresholding using predictive performance and MI. Furthermore, conceptual clustering achieves similar predictive performances without requiring parameterisation, and achieves this with lower computational overhead than thresholding using predictive performance and MI when the number of base models becomes large.
Enabling Reusable Physical Design Flows with Modular Flow Generators
Venue: arXiv cs.AR 2021
Year: 2021

Achieving high code reuse in physical design flows is challenging but increasingly necessary to build complex systems. Unfortunately, existing approaches based on parameterized Tcl generators support very limited reuse and struggle to preserve reusable code as designers customize flows for specific designs and technologies. We present a vision and framework based on modular flow generators that encapsulates coarse-grain and fine-grain reusable code in modular nodes and assembles them into complete flows. The key feature is a flow consistency and instrumentation layer embedded in Python, which supports mechanisms for rapid and early feedback on inconsistent composition. The approach gradually types the Tcl language and allows both automatic and user-annotated static assertion checks. We evaluate the design flows of successive generations of silicon prototypes designed in TSMC16, TSMC28, TSMC40, SKY130, and IBM180 technologies, showing how our approach can enable significant code reuse in future flows.
Deployment of Aerial Robots after a major fire of an industrial hall with hazardous substances, a report
Venue: arXiv cs.RO 2021
Year: 2021

This technical report is about the mission and the experience gained during the reconnaissance of an industrial hall with hazardous substances after a major fire in Berlin. During this operation, only UAVs and cameras were used to obtain information about the site and the building. First, a geo-referenced 3D model of the building was created in order to plan the entry into the hall. Subsequently, the UAVs were used to fly in the heavily damaged interior and take pictures from inside of the hall. A 360° camera mounted under the UAV was used to collect images of the surrounding area especially from sections that were difficult to fly into. Since the collected data set contained similar images as well as blurred images, it was cleaned from non-optimal images using visual SLAM, bundle adjustment and blur detection so that a 3D model and overviews could be calculated. It was shown that the emergency services were not able to extract the necessary information from the 3D model. Therefore, an interactive panorama viewer with links to other 360° images was implemented where the links to the other images depends on the semi dense point cloud and located camera positions of the visual SLAM algorithm so that the emergency forces could view the surroundings.
LiVLR: A Lightweight Visual-Linguistic Reasoning Framework for Video Question Answering
Venue: arXiv cs.CV 2021
Year: 2021

Video Question Answering (VideoQA), aiming to correctly answer the given question based on understanding multi-modal video content, is challenging due to the rich video content. From the perspective of video understanding, a good VideoQA framework needs to understand the video content at different semantic levels and flexibly integrate the diverse video content to distill question-related content. To this end, we propose a Lightweight Visual-Linguistic Reasoning framework named LiVLR. Specifically, LiVLR first utilizes the graph-based Visual and Linguistic Encoders to obtain multi-grained visual and linguistic representations. Subsequently, the obtained representations are integrated with the devised Diversity-aware Visual-Linguistic Reasoning module (DaVL). The DaVL considers the difference between the different types of representations and can flexibly adjust the importance of different types of representations when generating the question-related joint representation, which is an effective and general representation integration method. The proposed LiVLR is lightweight and shows its performance advantage on two VideoQA benchmarks, MRSVTT-QA and KnowIT VQA. Extensive ablation studies demonstrate the effectiveness of LiVLR key components.
MedRDF: A Robust and Retrain-Less Diagnostic Framework for Medical Pretrained Models Against Adversarial Attack
Venue: arXiv cs.CV 2021
Year: 2021

Deep neural networks are discovered to be non-robust when attacked by imperceptible adversarial examples, which is dangerous for it applied into medical diagnostic system that requires high reliability. However, the defense methods that have good effect in natural images may not be suitable for medical diagnostic tasks. The preprocessing methods (e.g., random resizing, compression) may lead to the loss of the small lesions feature in the medical image. Retraining the network on the augmented data set is also not practical for medical models that have already been deployed online. Accordingly, it is necessary to design an easy-to-deploy and effective defense framework for medical diagnostic tasks. In this paper, we propose a Robust and Retrain-Less Diagnostic Framework for Medical pretrained models against adversarial attack (i.e., MedRDF). It acts on the inference time of the pertained medical model. Specifically, for each test image, MedRDF firstly creates a large number of noisy copies of it, and obtains the output labels of these copies from the pretrained medical diagnostic model. Then, based on the labels of these copies, MedRDF outputs the final robust diagnostic result by majority voting. In addition to the diagnostic result, MedRDF produces the Robust Metric (RM) as the confidence of the result. Therefore, it is convenient and reliable to utilize MedRDF to convert pre-trained non-robust diagnostic models into robust ones. The experimental results on COVID-19 and DermaMNIST datasets verify the effectiveness of our MedRDF in improving the robustness of medical diagnostic models.
Multi-UAV Conflict Resolution with Graph Convolutional Reinforcement Learning
Venue: arXiv cs.LG 2021
Year: 2021

Safety is the primary concern when it comes to air traffic. In-flight safety between Unmanned Aircraft Vehicles (UAVs) is ensured through pairwise separation minima, utilizing conflict detection and resolution methods. Existing methods mainly deal with pairwise conflicts, however due to an expected increase in traffic density, encounters with more than two UAVs are likely to happen. In this paper, we model multi-UAV conflict resolution as a multi-agent reinforcement learning problem. We implement an algorithm based on graph neural networks where cooperative agents can communicate to jointly generate resolution maneuvers. The model is evaluated in scenarios with 3 and 4 present agents. Results show that agents are able to successfully solve the multi-UAV conflicts through a cooperative strategy.
On computable learning of continuous features
Venue: arXiv cs.LG 2021
Year: 2021

We introduce definitions of computable PAC learning for binary classification over computable metric spaces. We provide sufficient conditions for learners that are empirical risk minimizers (ERM) to be computable, and bound the strong Weihrauch degree of an ERM learner under more general conditions. We also give a presentation of a hypothesis class that does not admit any proper computable PAC learner with computable sample function, despite the underlying class being PAC learnable.
ILabel: Interactive Neural Scene Labelling
Venue: arXiv cs.CV 2021
Year: 2021

Joint representation of geometry, colour and semantics using a 3D neural field enables accurate dense labelling from ultra-sparse interactions as a user reconstructs a scene in real-time using a handheld RGB-D sensor. Our iLabel system requires no training data, yet can densely label scenes more accurately than standard methods trained on large, expensively labelled image datasets. Furthermore, it works in an 'open set' manner, with semantic classes defined on the fly by the user. ILabel's underlying model is a multilayer perceptron (MLP) trained from scratch in real-time to learn a joint neural scene representation. The scene model is updated and visualised in real-time, allowing the user to focus interactions to achieve efficient labelling. A room or similar scene can be accurately labelled into 10+ semantic categories with only a few tens of clicks. Quantitative labelling accuracy scales powerfully with the number of clicks, and rapidly surpasses standard pre-trained semantic segmentation methods. We also demonstrate a hierarchical labelling variant.
MUNet: Motion Uncertainty-aware Semi-supervised Video Object Segmentation
Venue: arXiv cs.CV 2021
Year: 2021

The task of semi-supervised video object segmentation (VOS) has been greatly advanced and state-of-the-art performance has been made by dense matching-based methods. The recent methods leverage space-time memory (STM) networks and learn to retrieve relevant information from all available sources, where the past frames with object masks form an external memory and the current frame as the query is segmented using the mask information in the memory. However, when forming the memory and performing matching, these methods only exploit the appearance information while ignoring the motion information. In this paper, we advocate the return of the \emph{motion information} and propose a motion uncertainty-aware framework (MUNet) for semi-supervised VOS. First, we propose an implicit method to learn the spatial correspondences between neighboring frames, building upon a correlation cost volume. To handle the challenging cases of occlusion and textureless regions during constructing dense correspondences, we incorporate the uncertainty in dense matching and achieve motion uncertainty-aware feature representation. Second, we introduce a motion-aware spatial attention module to effectively fuse the motion feature with the semantic feature. Comprehensive experiments on challenging benchmarks show that \textbf{\textit{using a small amount of data and combining it with powerful motion information can bring a significant performance boost}}. We achieve ${76.5\%}$ $\mathcal{J} \& \mathcal{F}$ only using DAVIS17 for training, which significantly outperforms the \textit{SOTA} methods under the low-data protocol. \textit{The code will be released.}
Online MAP Inference and Learning for Nonsymmetric Determinantal Point Processes
Venue: arXiv cs.LG 2021
Year: 2021

In this paper, we introduce the online and streaming MAP inference and learning problems for Non-symmetric Determinantal Point Processes (NDPPs) where data points arrive in an arbitrary order and the algorithms are constrained to use a single-pass over the data as well as sub-linear memory. The online setting has an additional requirement of maintaining a valid solution at any point in time. For solving these new problems, we propose algorithms with theoretical guarantees, evaluate them on several real-world datasets, and show that they give comparable performance to state-of-the-art offline algorithms that store the entire data in memory and take multiple passes over it.