Reviewerly/exHarmony huggingface dataset:
Multimodal Continual Learning with Sonographer Eye-Tracking in Fetal Ultrasound

Deep networks have been shown to achieve impressive accuracy for some medical image analysis tasks where large datasets and annotations are available. However, tasks involving learning over new sets of classes arriving over extended time is a different and difficult challenge due to the tendency of reduction in performance over old classes while adapting to new ones. Controlling such a 'forgetting' is vital for deployed algorithms to evolve with new arrivals of data incrementally. Usually, incremental learning approaches rely on expert knowledge in the form of manual annotations or active feedback. In this paper, we explore the role that other forms of expert knowledge might play in making deep networks in medical image analysis immune to forgetting over extended time. We introduce a novel framework for mitigation of this forgetting effect in deep networks considering the case of combining ultrasound video with point-of-gaze tracked for expert sonographers during model training. This is used along with a novel weighted distillation strategy to reduce the propagation of effects due to class imbalance.
BitTrain: Sparse Bitmap Compression for Memory-Efficient Training on the Edge

Training on the Edge enables neural networks to learn continuously from new data after deployment on memory-constrained edge devices. Previous work is mostly concerned with reducing the number of model parameters which is only beneficial for inference. However, memory footprint from activations is the main bottleneck for training on the edge. Existing incremental training methods fine-tune the last few layers sacrificing accuracy gains from re-training the whole model. In this work, we investigate the memory footprint of training deep learning models, and use our observations to propose BitTrain. In BitTrain, we exploit activation sparsity and propose a novel bitmap compression technique that reduces the memory footprint during training. We save the activations in our proposed bitmap compression format during the forward pass of the training, and restore them during the backward pass for the optimizer computations. The proposed method can be integrated seamlessly in the computation graph of modern deep learning frameworks. Our implementation is safe by construction, and has no negative impact on the accuracy of model training. Experimental results show up to 34% reduction in the memory footprint at a sparsity level of 50%. Further pruning during training results in more than 70% sparsity, which can lead to up to 56% reduction in memory footprint. BitTrain advances the efforts towards bringing more machine learning capabilities to edge devices. Our source code is available at https://github.com/scale-lab/BitTrain.
Parameterized Knowledge Transfer for Personalized Federated Learning

In recent years, personalized federated learning (pFL) has attracted increasing attention for its potential in dealing with statistical heterogeneity among clients. However, the state-of-the-art pFL methods rely on model parameters aggregation at the server side, which require all models to have the same structure and size, and thus limits the application for more heterogeneous scenarios. To deal with such model constraints, we exploit the potentials of heterogeneous model settings and propose a novel training framework to employ personalized models for different clients. Specifically, we formulate the aggregation procedure in original pFL into a personalized group knowledge transfer training algorithm, namely, KT-pFL, which enables each client to maintain a personalized soft prediction at the server side to guide the others' local training. KT-pFL updates the personalized soft prediction of each client by a linear combination of all local soft predictions using a knowledge coefficient matrix, which can adaptively reinforce the collaboration among clients who own similar data distribution. Furthermore, to quantify the contributions of each client to others' personalized training, the knowledge coefficient matrix is parameterized so that it can be trained simultaneously with the models. The knowledge coefficient matrix and the model parameters are alternatively updated in each round following the gradient descent way. Extensive experiments on various datasets (EMNIST, Fashion\_MNIST, CIFAR-10) are conducted under different settings (heterogeneous models and data distributions). It is demonstrated that the proposed framework is the first federated learning paradigm that realizes personalized model training via parameterized group knowledge transfer while achieving significant performance gain comparing with state-of-the-art algorithms.
Single-Item Fashion Recommender: Towards Cross-Domain Recommendations

Nowadays, recommender systems and search engines play an integral role in fashion e-commerce. Still, many challenges lie ahead, and this study tries to tackle some. This article first suggests a content-based fashion recommender system that uses a parallel neural network to take a single fashion item shop image as input and make in-shop recommendations by listing similar items available in the store. Next, the same structure is enhanced to personalize the results based on user preferences. This work then introduces a background augmentation technique that makes the system more robust to out-of-domain queries, enabling it to make street-to-shop recommendations using only a training set of catalog shop images. Moreover, the last contribution of this paper is a new evaluation metric for recommendation tasks called objective-guided human score. This method is an entirely customizable framework that produces interpretable, comparable scores from subjective evaluations of human scorers.
Deep Learning Image-To-Image Translation For Animation

This paper presents a novel application for animation of cartoon characters. The system consists of two components: (a) a generative adversarial network that is trained to generate comic characters from edge images, and (b) an edge detector that takes input from portraits and then generates comic characters based on the resulting edge images.
Impact of Target Word and Context on End-to-End Metonymy Detection

Metonymy is a figure of speech in which an entity is referred to by another related entity. The task of metonymy detection aims to distinguish metonymic tokens from literal ones. Until now, metonymy detection methods attempt to disambiguate only a single noun phrase in a sentence, typically location names or organization names. In this paper, we disambiguate every word in a sentence by reformulating metonymy detection as a sequence labeling task. We also investigate the impact of target word and context on metonymy detection. We show that the target word is less useful for detecting metonymy in our dataset. On the other hand, the entity types that are associated with domain-specific words in their context are easier to solve. This shows that the context words are much more relevant for detecting metonymy.
Biodiversity of Chordata in the Philippine Sea: A case study based on OBIS

Aims:The Philippine Sea is a habitat that produces many important biological resources.The purpose of this paper is to summarize baseline data on the distribution of marine Chordata biodiversity with threatened status, and provide suggestions for conservation planning for marine biodiversity in the Philippine Sea.Methods: In this paper, the Ocean Biogeographic Information System (OBIS) and the IUCN Red List of Threatened Species (IUCN Redlist) were used to collect and assess current biodiversity data of Chordata in the Philippine Sea.A list of Chordata species with threatened status in the Philippine Sea was compiled and sorted.A visualization of the spatial distribution of biodiversity at different taxonomic and threatened levels in this area was also produced.The relationships between the richness at varying taxonomic levels and environmental factors, such as net primary productivity, surface sea temperature, and water depth in the Philippine Sea were also explored by correlation analysis. Results:The results showed that the 2,876 species in the phylum of Chordata in this region belong to 11 classes, 56 orders, 320 families, 1,171 genera.The richness at varying taxonomic levels was high in the waters near the Philippine Islands, Taiwan Island, Japanese Island, the Kyushu Palau ridge, and Mariana Islands, while the richness in the central •研究报告•
Verifying liquidity of recursive Bitcoin contracts

Smart contracts - computer protocols that regulate the exchange of crypto-assets in trustless environments - have become popular with the spread of blockchain technologies. A landmark security property of smart contracts is liquidity: in a non-liquid contract, it may happen that some assets remain frozen, i.e. not redeemable by anyone. The relevance of this issue is witnessed by recent liquidity attacks to Ethereum, which have frozen hundreds of USD millions. We address the problem of verifying liquidity on BitML, a DSL for smart contracts with a secure compiler to Bitcoin, featuring primitives for currency transfers, contract renegotiation and consensual recursion. Our main result is a verification technique for liquidity. We first transform the infinite-state semantics of BitML into a finite-state one, which focusses on the behaviour of a chosen set of contracts, abstracting from the moves of the context. With respect to the chosen contracts, this abstraction is sound, i.e. if the abstracted contract is liquid, then also the concrete one is such. We then verify liquidity by model-checking the finite-state abstraction. We implement a toolchain that automatically verifies liquidity of BitML contracts and compiles them to Bitcoin, and we assess it through a benchmark of representative contracts.
Deep Relation Learning for Regression and Its Application to Brain Age Estimation

Most deep learning models for temporal regression directly output the estimation based on single input images, ignoring the relationships between different images. In this paper, we propose deep relation learning for regression, aiming to learn different relations between a pair of input images. Four non-linear relations are considered: "cumulative relation", "relative relation", "maximal relation" and "minimal relation". These four relations are learned simultaneously from one deep neural network which has two parts: feature extraction and relation regression. We use an efficient convolutional neural network to extract deep features from the pair of input images and apply a Transformer for relation learning. The proposed method is evaluated on a merged dataset with 6,049 subjects with ages of 0-97 years using 5-fold cross-validation for the task of brain age estimation. The experimental results have shown that the proposed method achieved a mean absolute error (MAE) of 2.38 years, which is lower than the MAEs of 8 other state-of-the-art algorithms with statistical significance (p$<$0.05) in paired T-test (two-side).
Quantum computation with cat qubits

These are the lecture notes from the 2019 Les Houches Summer School on "Quantum Information Machines". After a brief introduction to quantum error correction and bosonic codes, we focus on the case of cat qubits stabilized by a nonlinear multi-photon driven dissipation process. We argue that such a system can be seen as a self-correcting qubit where bit-flip errors are robustly and exponentially suppressed. Next, we provide some experimental directions to engineer such a multi-photon driven dissipation process with superconducting circuits. Finally, we analyze various logical gates that can be implemented without re-introducing bit-flip errors. This set of bias-preserving gates pave the way towards a hardware-efficient and fault-tolerant quantum processor.
ViT-FOD: A Vision Transformer based Fine-grained Object Discriminator

Recently, several Vision Transformer (ViT) based methods have been proposed for Fine-Grained Visual Classification (FGVC).These methods significantly surpass existing CNN-based ones, demonstrating the effectiveness of ViT in FGVC tasks.However, there are some limitations when applying ViT directly to FGVC.First, ViT needs to split images into patches and calculate the attention of every pair, which may result in heavy redundant calculation and unsatisfying performance when handling fine-grained images with complex background and small objects.Second, a standard ViT only utilizes the class token in the final layer for classification, which is not enough to extract comprehensive fine-grained information. To address these issues, we propose a novel ViT based fine-grained object discriminator for FGVC tasks, ViT-FOD for short. Specifically, besides a ViT backbone, it further introduces three novel components, i.e, Attention Patch Combination (APC), Critical Regions Filter (CRF), and Complementary Tokens Integration (CTI). Thereinto, APC pieces informative patches from two images to generate a new image so that the redundant calculation can be reduced. CRF emphasizes tokens corresponding to discriminative regions to generate a new class token for subtle feature learning. To extract comprehensive information, CTI integrates complementary information captured by class tokens in different ViT layers. We conduct comprehensive experiments on widely used datasets and the results demonstrate that ViT-FOD is able to achieve state-of-the-art performance.
Same Neurons, Different Languages: Probing Morphosyntax in Multilingual Pre-trained Models

The success of multilingual pre-trained models is underpinned by their ability to learn representations shared by multiple languages even in absence of any explicit supervision. However, it remains unclear how these models learn to generalise across languages. In this work, we conjecture that multilingual pre-trained models can derive language-universal abstractions about grammar. In particular, we investigate whether morphosyntactic information is encoded in the same subset of neurons in different languages. We conduct the first large-scale empirical study over 43 languages and 14 morphosyntactic categories with a state-of-the-art neuron-level probe. Our findings show that the cross-lingual overlap between neurons is significant, but its extent may vary across categories and depends on language proximity and pre-training data size.
Machine Learning Sensors

Machine learning sensors represent a paradigm shift for the future of embedded machine learning applications. Current instantiations of embedded machine learning (ML) suffer from complex integration, lack of modularity, and privacy and security concerns from data movement. This article proposes a more data-centric paradigm for embedding sensor intelligence on edge devices to combat these challenges. Our vision for "sensor 2.0" entails segregating sensor input data and ML processing from the wider system at the hardware level and providing a thin interface that mimics traditional sensors in functionality. This separation leads to a modular and easy-to-use ML sensor device. We discuss challenges presented by the standard approach of building ML processing into the software stack of the controlling microprocessor on an embedded system and how the modularity of ML sensors alleviates these problems. ML sensors increase privacy and accuracy while making it easier for system builders to integrate ML into their products as a simple component. We provide examples of prospective ML sensors and an illustrative datasheet as a demonstration and hope that this will build a dialogue to progress us towards sensor 2.0.
Understanding convolution on graphs via energies

Graph Neural Networks (GNNs) typically operate by message-passing, where the state of a node is updated based on the information received from its neighbours. Most message-passing models act as graph convolutions, where features are mixed by a shared, linear transformation before being propagated over the edges. On node-classification tasks, graph convolutions have been shown to suffer from two limitations: poor performance on heterophilic graphs, and over-smoothing. It is common belief that both phenomena occur because such models behave as low-pass filters, meaning that the Dirichlet energy of the features decreases along the layers incurring a smoothing effect that ultimately makes features no longer distinguishable. In this work, we rigorously prove that simple graph-convolutional models can actually enhance high frequencies and even lead to an asymptotic behaviour we refer to as over-sharpening, opposite to over-smoothing. We do so by showing that linear graph convolutions with symmetric weights minimize a multi-particle energy that generalizes the Dirichlet energy; in this setting, the weight matrices induce edge-wise attraction (repulsion) through their positive (negative) eigenvalues, thereby controlling whether the features are being smoothed or sharpened. We also extend the analysis to non-linear GNNs, and demonstrate that some existing time-continuous GNNs are instead always dominated by the low frequencies. Finally, we validate our theoretical findings through ablations and real-world experiments.
FedIIC: Towards Robust Federated Learning for Class-Imbalanced Medical Image Classification

Federated learning (FL), training deep models from decentralized data without privacy leakage, has shown great potential in medical image computing recently. However, considering the ubiquitous class imbalance in medical data, FL can exhibit performance degradation, especially for minority classes (e.g. rare diseases). Existing methods towards this problem mainly focus on training a balanced classifier to eliminate class prior bias among classes, but neglect to explore better representation to facilitate classification performance. In this paper, we present a privacy-preserving FL method named FedIIC to combat class imbalance from two perspectives: feature learning and classifier learning. In feature learning, two levels of contrastive learning are designed to extract better class-specific features with imbalanced data in FL. In classifier learning, per-class margins are dynamically set according to real-time difficulty and class priors, which helps the model learn classes equally. Experimental results on publicly-available datasets demonstrate the superior performance of FedIIC in dealing with both real-world and simulated multi-source medical imaging data under class imbalance. Code is available at https://github.com/wnn2000/FedIIC.
Towards Scalable Model Checking of Reflective Systems via Labeled Transition Systems

Reflection is a technique that enables a system to inspect or change its structure and/or behavior at runtime. It is a key enabler of many techniques for developing systems that have to function despite rapidly changing requirements and environments. A crucial issue in developing reflective systems is to ensure the correctness of their behaviors, because object-level behaviors are affected by metalevel behaviors. In this paper, we present an extended labeled transition system (LTS), which we call a metalevel LTS (MLTS), that supports data representation of another LTS for use in modeling a reflective tower. We show that two of the existing state reduction techniques for an LTS ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">symmetry reduction</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">divergence-sensitive stutter bisimulation</i> ) are also applicable to an MLTS. Then, we introduce two strategies for implementing an MLTS model in Promela, thereby enabling verification with the SPIN model checker. We also present case studies of applying MLTSs to two reflection applications: self-adaptation of a reconnaissance robot system, and dynamic evolution of an Internet-of-things (IoT) system. The case studies demonstrate the applicability of our approach and its scalability improvement through the state reduction techniques.
Deep Variational Quantum Eigensolver: a divide-and-conquer method for solving a larger problem with smaller size quantum computers

We propose a divide-and-conquer method for the quantum-classical hybrid algorithm to solve larger problems with small-scale quantum computers. Specifically, we concatenate a variational quantum eigensolver (VQE) with a reduction in the system dimension, where the interactions between divided subsystems are taken as an effective Hamiltonian expanded by the reduced basis. Then the effective Hamiltonian is further solved by VQE, which we call deep VQE. Deep VQE allows us to apply quantum-classical hybrid algorithms on small-scale quantum computers to large systems with strong intra-subsystem interactions and weak inter-subsystem interactions, or strongly correlated spin models on large regular lattices. As proof-of-principle numerical demonstrations, we use the proposed method for quasi one-dimensional models, including one-dimensionally coupled 12-qubit Heisenberg anti-ferromagnetic models on Kagome lattices as well as two-dimensional Heisenberg anti-ferromagnetic models on square lattices. The largest problem size of 64 qubits is solved by simulating 20-qubit quantum computers with a reasonably good accuracy ~ a few %. The proposed scheme enables us to handle the problems of >1000 qubits by concatenating VQEs with a few tens of qubits. While it is unclear how accurate ground state energy can be obtained for such a large system, our numerical results on a 64-qubit system suggest that deep VQE provides a good approximation (discrepancy within a few percent) and has a room for further improvement. Therefore, deep VQE provides us a promising pathway to solve practically important problems on noisy intermediate-scale quantum computers.
Fuzzm: Finding Memory Bugs through Binary-Only Instrumentation and Fuzzing of WebAssembly

WebAssembly binaries are often compiled from memory-unsafe languages, such as C and C++. Because of WebAssembly's linear memory and missing protection features, e.g., stack canaries, source-level memory vulnerabilities are exploitable in compiled WebAssembly binaries, sometimes even more easily than in native code. This paper addresses the problem of detecting such vulnerabilities through the first binary-only fuzzer for WebAssembly. Our approach, called Fuzzm, combines canary instrumentation to detect overflows and underflows on the stack and the heap, an efficient coverage instrumentation, a WebAssembly VM, and the input generation algorithm of the popular AFL fuzzer. Besides as an oracle for fuzzing, our canaries also serve as a stand-alone binary hardening technique to prevent the exploitation of vulnerable binaries in production. We evaluate Fuzzm with 28 real-world WebAssembly binaries, some compiled from source and some found in the wild without source code. The fuzzer explores thousands of execution paths, triggers dozens of crashes, and performs hundreds of program executions per second. When used for binary hardening, the approach prevents previously published exploits against vulnerable WebAssembly binaries while imposing low runtime overhead.