OpenRaiser/Intern-Atlas huggingface dataset:
Building Domain-Specific Machine Learning Workflows: A Conceptual Framework for the State-of-the-Practice
Venue: arXiv cs.SE 2022
Year: 2022

Domain experts are increasingly employing machine learning to solve their domain-specific problems. This article presents six key challenges that a domain expert faces in transforming their problem into a computational workflow, and then into an executable implementation. These challenges arise out of our conceptual framework which presents the "route" of options that a domain expert may choose to take while developing their solution. To ground our conceptual framework in the state-of-the-practice, this article discusses a selection of available textual and graphical workflow systems and their support for these six challenges. Case studies from the literature in various domains are also examined to highlight the tools used by the domain experts as well as a classification of the domain-specificity and machine learning usage of their problem, workflow, and implementation. The state-of-the-practice informs our discussion of the six key challenges, where we identify which challenges are not sufficiently addressed by available tools. We also suggest possible research directions for software engineering researchers to increase the automation of these tools and disseminate best-practice techniques between software engineering and various scientific domains.
Counterfactual Inference of Second Opinions
Venue: arXiv cs.LG 2022
Year: 2022

Automated decision support systems that are able to infer second opinions from experts can potentially facilitate a more efficient allocation of resources; they can help decide when and from whom to seek a second opinion. In this paper, we look at the design of this type of support systems from the perspective of counterfactual inference. We focus on a multiclass classification setting and first show that, if experts make predictions on their own, the underlying causal mechanism generating their predictions needs to satisfy a desirable set invariant property. Further, we show that, for any causal mechanism satisfying this property, there exists an equivalent mechanism where the predictions by each expert are generated by independent sub-mechanisms governed by a common noise. This motivates the design of a set invariant Gumbel-Max structural causal model where the structure of the noise governing the sub-mechanisms underpinning the model depends on an intuitive notion of similarity between experts which can be estimated from data. Experiments on both synthetic and real data show that our model can be used to infer second opinions more accurately than its non-causal counterpart.
Learning Representation for Bayesian Optimization with Collision-free Regularization
Venue: arXiv cs.LG 2022
Year: 2022

Bayesian optimization has been challenged by datasets with large-scale, high-dimensional, and non-stationary characteristics, which are common in real-world scenarios. Recent works attempt to handle such input by applying neural networks ahead of the classical Gaussian process to learn a latent representation. We show that even with proper network design, such learned representation often leads to collision in the latent space: two points with significantly different observations collide in the learned latent space, leading to degraded optimization performance. To address this issue, we propose LOCo, an efficient deep Bayesian optimization framework which employs a novel regularizer to reduce the collision in the learned latent space and encourage the mapping from the latent space to the objective value to be Lipschitz continuous. LOCo takes in pairs of data points and penalizes those too close in the latent space compared to their target space distance. We provide a rigorous theoretical justification for LOCo by inspecting the regret of this dynamic-embedding-based Bayesian optimization algorithm, where the neural network is iteratively retrained with the regularizer. Our empirical results demonstrate the effectiveness of LOCo on several synthetic and real-world benchmark Bayesian optimization tasks.
Graph Flow: Cross-layer Graph Flow Distillation for Dual Efficient Medical Image Segmentation
Venue: arXiv cs.CV 2022
Year: 2022

With the development of deep convolutional neural networks, medical image segmentation has achieved a series of breakthroughs in recent years. However, the high-performance convolutional neural networks always mean numerous parameters and high computation costs, which will hinder the applications in clinical scenarios. Meanwhile, the scarceness of large-scale annotated medical image datasets further impedes the application of high-performance networks. To tackle these problems, we propose Graph Flow, a comprehensive knowledge distillation framework, for both network-efficiency and annotation-efficiency medical image segmentation. Specifically, our core Graph Flow Distillation transfer the essence of cross-layer variations from a well-trained cumbersome teacher network to a non-trained compact student network. In addition, an unsupervised Paraphraser Module is integrated to purify the knowledge of the teacher network, which is also beneficial for the stabilization of training procedure. Furthermore, we build a unified distillation framework by integrating the adversarial distillation and the vanilla logits distillation, which can further refine the final predictions of the compact network. With different teacher networks (conventional convolutional architecture or prevalent transformer architecture) and student networks, we conduct extensive experiments on four medical image datasets with different modalities (Gastric Cancer, Synapse, BUSI, and CVC-ClinicDB).We demonstrate the prominent ability of our method which achieves competitive performance on these datasets. Moreover, we demonstrate the effectiveness of our Graph Flow through a novel semi-supervised paradigm for dual efficient medical image segmentation. Our code will be available at Graph Flow.
MPAF: Model Poisoning Attacks to Federated Learning based on Fake Clients
Venue: arXiv cs.CR 2022
Year: 2022

Existing model poisoning attacks to federated learning assume that an attacker has access to a large fraction of compromised genuine clients. However, such assumption is not realistic in production federated learning systems that involve millions of clients. In this work, we propose the first Model Poisoning Attack based on Fake clients called MPAF. Specifically, we assume the attacker injects fake clients to a federated learning system and sends carefully crafted fake local model updates to the cloud server during training, such that the learnt global model has low accuracy for many indiscriminate test inputs. Towards this goal, our attack drags the global model towards an attacker-chosen base model that has low accuracy. Specifically, in each round of federated learning, the fake clients craft fake local model updates that point to the base model and scale them up to amplify their impact before sending them to the cloud server. Our experiments show that MPAF can significantly decrease the test accuracy of the global model, even if classical defenses and norm clipping are adopted, highlighting the need for more advanced defenses.
Decoupled Knowledge Distillation
Venue: arXiv cs.CV 2022
Year: 2022

State-of-the-art distillation methods are mainly based on distilling deep features from intermediate layers, while the significance of logit distillation is greatly overlooked. To provide a novel viewpoint to study logit distillation, we reformulate the classical KD loss into two parts, i.e., target class knowledge distillation (TCKD) and non-target class knowledge distillation (NCKD). We empirically investigate and prove the effects of the two parts: TCKD transfers knowledge concerning the "difficulty" of training samples, while NCKD is the prominent reason why logit distillation works. More importantly, we reveal that the classical KD loss is a coupled formulation, which (1) suppresses the effectiveness of NCKD and (2) limits the flexibility to balance these two parts. To address these issues, we present Decoupled Knowledge Distillation (DKD), enabling TCKD and NCKD to play their roles more efficiently and flexibly. Compared with complex feature-based methods, our DKD achieves comparable or even better results and has better training efficiency on CIFAR-100, ImageNet, and MS-COCO datasets for image classification and object detection tasks. This paper proves the great potential of logit distillation, and we hope it will be helpful for future research. The code is available at https://github.com/megvii-research/mdistiller.
CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals
Venue: arXiv cs.CL 2022
Year: 2022

We propose a framework to modularize the training of neural language models that use diverse forms of sentence-external context (including metadata) by eliminating the need to jointly train sentence-external and within-sentence encoders. Our approach, contextual universal embeddings (CUE), trains LMs on one set of context, such as date and author, and adapts to novel metadata types, such as article title, or previous sentence. The model consists of a pretrained neural sentence LM, a BERT-based context encoder, and a masked transformer decoder that estimates LM probabilities using sentence-internal and sentence-external information. When context or metadata are unavailable, our model learns to combine contextual and sentence-internal information using noisy oracle unigram embeddings as a proxy. Real contextual information can be introduced later and used to adapt a small number of parameters that map contextual data into the decoder's embedding space. We validate the CUE framework on a NYTimes text corpus with multiple metadata types, for which the LM perplexity can be lowered from 36.6 to 27.4 by conditioning on context. Bootstrapping a contextual LM with only a subset of the context/metadata during training retains 85\% of the achievable gain. Training the model initially with proxy context retains 67% of the perplexity gain after adapting to real context. Furthermore, we can swap one type of pretrained sentence LM for another without retraining the context encoders, by only adapting the decoder model. Overall, we obtain a modular framework that allows incremental, scalable training of context-enhanced LMs.
A Continual Learning Framework for Adaptive Defect Classification and Inspection
Venue: arXiv cs.CV 2022
Year: 2022

Machine-vision-based defect classification techniques have been widely adopted for automatic quality inspection in manufacturing processes. This article describes a general framework for classifying defects from high volume data batches with efficient inspection of unlabelled samples. The concept is to construct a detector to identify new defect types, send them to the inspection station for labelling, and dynamically update the classifier in an efficient manner that reduces both storage and computational needs imposed by data samples of previously observed batches. Both a simulation study on image classification and a case study on surface defect detection via 3D point clouds are performed to demonstrate the effectiveness of the proposed method.
Example Perplexity
Venue: arXiv cs.LG 2022
Year: 2022

Some examples are easier for humans to classify than others. The same should be true for deep neural networks (DNNs). We use the term example perplexity to refer to the level of difficulty of classifying an example. In this paper, we propose a method to measure the perplexity of an example and investigate what factors contribute to high example perplexity. The related codes and resources are available at https://github.com/vaynexie/Example-Perplexity.
QUBOs for Sorting Lists and Building Trees
Venue: arXiv cs.DS 2022
Year: 2022

We show that the fundamental tasks of sorting lists and building search trees or heaps can be modeled as quadratic unconstrained binary optimization problems (QUBOs). The idea is to understand these tasks as permutation problems and to devise QUBOs whose solutions represent appropriate permutation matrices. We discuss how to construct such QUBOs and how to solve them using Hopfield nets or adiabatic) quantum computing. In short, we show that neurocomputing methods or quantum computers can solve problems usually associated with abstract data structures.
"It Feels Like Taking a Gamble": Exploring Perceptions, Practices, and Challenges of Using Makeup and Cosmetics for People with Visual Impairments
Venue: arXiv cs.HC 2022
Year: 2022

Makeup and cosmetics offer the potential for self-expression and the reshaping of social roles for visually impaired people. However, there exist barriers to conducting a beauty regime because of the reliance on visual information and color variances in makeup. We present a content analysis of 145 YouTube videos to demonstrate visually impaired individuals' unique practices before, during, and after doing makeup. Based on the makeup practices, we then conducted semi-structured interviews with 12 visually impaired people to discuss their perceptions of and challenges with the makeup process in more depth. Overall, through our findings and discussion, we present novel perceptions of makeup from visually impaired individuals (e.g., broader representations of blindness and beauty). The existing challenges provide opportunities for future research to address learning barriers, insufficient feedback, and physical and environmental barriers, making the experience of doing makeup more accessible to people with visual impairments.
Blockchain as privacy and security solution for smart environments: A Survey
Venue: arXiv cs.CR 2022
Year: 2022

Blockchain was always associated with Bitcoin, cryptocurrencies, and digital asset trading. However, its benefits are far beyond that. It supports technologies like the Internet-of-Things (IoT) to pave the way for futuristic smart environments, like smart homes, smart transportation, smart energy trading, smart industries, smart supply chains, and more. To enable these environments, IoT devices, machines, appliances, and vehicles, need to intercommunicate without the need for centralized trusted parties. Blockchain replaces these trusted parties in such trustless environments. It provides security enforcement, privacy assurance, authentication, and other key features to IoT ecosystems. Besides IoT-Blockchain integration, other technologies add more benefits that attract the research community. Software-Defined Networking (SDN), Fog, Edge, and Cloud Computing technologies, for example, play a key role in enabling realistic IoT applications. Moreover, the integration of Artificial Intelligence (AI) provides smart, dynamic, and autonomous decision-making capabilities for IoT devices in smart environments. To push the research further in this domain, we provide in this paper a comprehensive survey that includes state-of-the-art technological integration, challenges, and solutions for smart environments, and the role of these technologies as the building blocks of such smart environments. We also demonstrate how the level of integration between these technologies has increased over the years, which brings us closer to the futuristic view of smart environments. We further discuss the current need to provide general-purpose Blockchain platforms that can adapt to unique design requirements of different applications and solutions. Finally, we provide a simplified architecture of futuristic smart environments that integrate these technologies, showing the advantage of such integration.
One Bad Apple Can Spoil Your IPv6 Privacy
Venue: arXiv cs.CR 2022
Year: 2022

IPv6 is being more and more adopted, in part to facilitate the millions of smart devices that have already been installed at home. Unfortunately, we find that the privacy of a substantial fraction of end-users is still at risk, despite the efforts by ISPs and electronic vendors to improve end-user security, e.g., by adopting prefix rotation and IPv6 privacy extensions. By analyzing passive data from a large ISP, we find that around 19% of end-users' privacy can be at risk. When we investigate the root causes, we notice that a single device at home that encodes its MAC address into the IPv6 address can be utilized as a tracking identifier for the entire end-user prefix -- even if other devices use IPv6 privacy extensions. Our results show that IoT devices contribute the most to this privacy leakage and, to a lesser extent, personal computers and mobile devices. To our surprise, some of the most popular IoT manufacturers have not yet adopted privacy extensions that could otherwise mitigate this privacy risk. Finally, we show that third-party providers, e.g., hypergiants, can track up to 17% of subscriber lines in our study.
Detecting silent data corruptions in the wild
Venue: arXiv cs.AR 2022
Year: 2022

Silent Errors within hardware devices occur when an internal defect manifests in a part of the circuit which does not have check logic to detect the incorrect circuit operation. The results of such a defect can range from flipping a single bit in a single data value, up to causing the software to execute the wrong instructions. Silent data corruptions (SDC) in hardware impact computational integrity for large-scale applications. Manifestations of silent errors are accelerated by datapath variations, temperature variance, and age, among other silicon factors. These errors do not leave any record or trace in system logs. As a result, silent errors stay undetected within workloads, and their effects can propagate across several services, causing problems to appear in systems far removed from the original defect. In this paper, we describe testing strategies to detect silent data corruptions within a large scale infrastructure. Given the challenging nature of the problem, we experimented with different methods for detection and mitigation. We compare and contrast two such approaches - 1. Fleetscanner (out-of-production testing) and 2. Ripple (in-production testing).We evaluate the infrastructure tradeoffs associated with the silicon testing funnel across 3+ years of production experience.
Semantic-diversity transfer network for generalized zero-shot learning via inner disagreement based OOD detector
Venue: arXiv cs.CV 2022
Year: 2022

Zero-shot learning (ZSL) aims to recognize objects from unseen classes, where the kernel problem is to transfer knowledge from seen classes to unseen classes by establishing appropriate mappings between visual and semantic features. The knowledge transfer in many existing works is limited mainly due to the facts that 1) the widely used visual features are global ones but not totally consistent with semantic attributes; 2) only one mapping is learned in existing works, which is not able to effectively model diverse visual-semantic relations; 3) the bias problem in the generalized ZSL (GZSL) could not be effectively handled. In this paper, we propose two techniques to alleviate these limitations. Firstly, we propose a Semantic-diversity transfer Network (SetNet) addressing the first two limitations, where 1) a multiple-attention architecture and a diversity regularizer are proposed to learn multiple local visual features that are more consistent with semantic attributes and 2) a projector ensemble that geometrically takes diverse local features as inputs is proposed to model visual-semantic relations from diverse local perspectives. Secondly, we propose an inner disagreement based domain detection module (ID3M) for GZSL to alleviate the third limitation, which picks out unseen-class data before class-level classification. Due to the absence of unseen-class data in training stage, ID3M employs a novel self-contained training scheme and detects out unseen-class data based on a designed inner disagreement criterion. Experimental results on three public datasets demonstrate that the proposed SetNet with the explored ID3M achieves a significant improvement against $30$ state-of-the-art methods.
Complex Network Analysis of the Bitcoin Transaction Network
Venue: arXiv cs.SI 2022
Year: 2022

In this brief, we conduct a complex-network analysis of the Bitcoin transaction network. In particular, we design a new sampling method, namely random walk with flying-back (RWFB), to conduct effective data sampling. We then conduct a comprehensive analysis of the Bitcoin network in terms of the degree distribution, clustering coefficient, the shortest-path length, connected component, centrality, assortativity, and the rich-club coefficient. We obtain several important observations including the small-world phenomenon, multi-center status, preferential attachment, and non-rich-club effect of the current network. This work brings up an in-depth understanding of the current Bitcoin blockchain network and offers implications for future directions in malicious activity and fraud detection in cryptocurrency blockchain networks.
HybridNets: End-to-End Perception Network
Venue: arXiv cs.CV 2022
Year: 2022

End-to-end Network has become increasingly important in multi-tasking. One prominent example of this is the growing significance of a driving perception system in autonomous driving. This paper systematically studies an end-to-end perception network for multi-tasking and proposes several key optimizations to improve accuracy. First, the paper proposes efficient segmentation head and box/class prediction networks based on weighted bidirectional feature network. Second, the paper proposes automatically customized anchor for each level in the weighted bidirectional feature network. Third, the paper proposes an efficient training loss function and training strategy to balance and optimize network. Based on these optimizations, we have developed an end-to-end perception network to perform multi-tasking, including traffic object detection, drivable area segmentation and lane detection simultaneously, called HybridNets, which achieves better accuracy than prior art. In particular, HybridNets achieves 77.3 mean Average Precision on Berkeley DeepDrive Dataset, outperforms lane detection with 31.6 mean Intersection Over Union with 12.83 million parameters and 15.6 billion floating-point operations. In addition, it can perform visual perception tasks in real-time and thus is a practical and accurate solution to the multi-tasking problem. Code is available at https://github.com/datvuthanh/HybridNets.
Collision Avoidance of 3-Dimensional Objects in Dynamic Environments
Venue: arXiv cs.RO 2022
Year: 2022

Achieving collision avoidance between moving objects is an important objective while determining robot trajectories. In performing collision avoidance maneuvers, the relative shapes of the objects play an important role. The literature largely models the shapes of the objects as spheres, and this can make the avoidance maneuvers very conservative, especially when the objects are of elongated shape and/or non-convex. In this paper, we model the shapes of the objects using suitable combinations of ellipsoids and one-sheeted/two-sheeted hyperboloids, and employ a collision cone approach to achieve collision avoidance. We present a method to construct the 3-D collision cone, and present simulation results demonstrating the working of the collision avoidance laws.