OpenRaiser/Intern-Atlas huggingface dataset:
An exploration of asocial and social learning in the evolution of variable-length structures
Venue: arXiv cs.NE 2021
Year: 2021

We wish to explore the contribution that asocial and social learning might play as a mechanism for self-adaptation in the search for variable-length structures by an evolutionary algorithm. An extremely challenging, yet simple to understand problem landscape is adopted where the probability of randomly finding a solution is approximately one in a trillion. A number of learning mechanisms operating on variable-length structures are implemented and their performance analysed. The social learning setup, which combines forms of both social and asocial learning in combination with evolution is found to be most performant, while the setups exclusively adopting evolution are incapable of finding solutions.
Context-Adaptive Document-Level Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Most existing document-level neural machine translation (NMT) models leverage a fixed number of the previous or all global source sentences to handle the context-independent problem in standard NMT. However, the translating of each source sentence benefits from various sizes of context, and inappropriate context may harm the translation performance. In this work, we introduce a data-adaptive method that enables the model to adopt the necessary and useful context. Specifically, we introduce a light predictor into two document-level translation models to select the explicit context. Experiments demonstrate the proposed approach can significantly improve the performance over the previous methods with a gain up to 1.99 BLEU points.
Membership Inference Attack Susceptibility of Clinical Language Models
Venue: arXiv cs.CL 2021
Year: 2021

Deep Neural Network (DNN) models have been shown to have high empirical privacy leakages. Clinical language models (CLMs) trained on clinical data have been used to improve performance in biomedical natural language processing tasks. In this work, we investigate the risks of training-data leakage through white-box or black-box access to CLMs. We design and employ membership inference attacks to estimate the empirical privacy leaks for model architectures like BERT and GPT2. We show that membership inference attacks on CLMs lead to non-trivial privacy leakages of up to 7%. Our results show that smaller models have lower empirical privacy leakages than larger ones, and masked LMs have lower leakages than auto-regressive LMs. We further show that differentially private CLMs can have improved model utility on clinical domain while ensuring low empirical privacy leakage. Lastly, we also study the effects of group-level membership inference and disease rarity on CLM privacy leakages.
Automatic Guided Vehicles System and Its Coordination Control for Containers Terminal Logistics Application
Venue: arXiv cs.RO 2021
Year: 2021

Automatic Guided Vehicle (AGV) has been widely applied in automatic logistics system because it provides flexibility and efficiency. This paper addresses review and design of multi AGVs system (MAGS) and its coordination control to be implemented at containers terminal logistics systems. It focuses on design of electric AGV and supervisory control to coordinate multi AGVs. Firstly, related previous works by other researchers are reviewed including containers terminal system, application of AGV in logistics systems and multi mobile robots control systems. Secondly, application of AGV system in containers terminal is proposed including AGV design specification, AGV control method, multi AGV system supervisory control method, and containers terminal port model. Conclusions have been obtained as follows: an electric AGV may be developed using electric motors with total power between 700-900KW and speed of 228 rpm; a MAGVS is proposed based on free ranging navigation system using GPS, digital compass and rotary encoder sensors; path planning, coordination and traffic management control is conducted by the supervisory controller based on priority based coordinative (PBC) control algorithm; a model of MAGVS for containers terminal logistics system can be developed using the navigation system and PBC control algorithm.
In Defense of the Paper
Venue: arXiv cs.LG 2021
Year: 2021

The machine learning publication process is broken, of that there can be no doubt. Many of these flaws are attributed to the current workflow: LaTeX to PDF to reviewers to camera ready PDF. This has understandably resulted in the desire for new forms of publications; ones that can increase inclusively, accessibility and pedagogical strength. However, this venture fails to address the origins of these inadequacies in the contemporary paper workflow. The paper, being the basic unit of academic research, is merely how problems in the publication and research ecosystem manifest; but is not itself responsible for them. Not only will simply replacing or augmenting papers with different formats not fix existing problems; when used as a band-aid without systemic changes, will likely exacerbate the existing inequities. In this work, we argue that the root cause of hindrances in the accessibility of machine learning research lies not in the paper workflow but within the misaligned incentives behind the publishing and research processes. We discuss these problems and argue that the paper is the optimal workflow. We also highlight some potential solutions for the incentivization problems.
Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
Venue: arXiv cs.DC 2021
Year: 2021

Stochastic Gradient Descent (SGD) has become the de facto way to train deep neural networks in distributed clusters. A critical factor in determining the training throughput and model accuracy is the choice of the parameter synchronization protocol. For example, while Bulk Synchronous Parallel (BSP) often achieves better converged accuracy, the corresponding training throughput can be negatively impacted by stragglers. In contrast, Asynchronous Parallel (ASP) can have higher throughput, but its convergence and accuracy can be impacted by stale gradients. To improve the performance of synchronization protocol, recent work often focuses on designing new protocols with a heavy reliance on hard-to-tune hyper-parameters. In this paper, we design a hybrid synchronization approach that exploits the benefits of both BSP and ASP, i.e., reducing training time while simultaneously maintaining the converged accuracy. Based on extensive empirical profiling, we devise a collection of adaptive policies that determine how and when to switch between synchronization protocols. Our policies include both offline ones that target recurring jobs and online ones for handling transient stragglers. We implement the proposed policies in a prototype system, called Sync-Switch, on top of TensorFlow, and evaluate the training performance with popular deep learning models and datasets. Our experiments show that Sync-Switch achieves up to 5.13X throughput speedup and similar converged accuracy when comparing to BSP. Further, we observe that Sync-Switch achieves 3.8% higher converged accuracy with just 1.23X the training time compared to training with ASP. Moreover, Sync-Switch can be used in settings when training with ASP leads to divergence errors. Sync-Switch achieves all of these benefits with very low overhead, e.g., the framework overhead can be as low as 1.7% of the total training time.
Verified Double Sided Auctions for Financial Markets
Venue: arXiv cs.GT 2021
Year: 2021

Double sided auctions are widely used in financial markets to match demand and supply. Prior works on double sided auctions have focused primarily on single quantity trade requests. We extend various notions of double sided auctions to incorporate multiple quantity trade requests and provide fully formalized matching algorithms for double sided auctions with their correctness proofs. We establish new uniqueness theorems that enable automatic detection of violations in an exchange program by comparing its output with that of a verified program. All proofs are formalized in the Coq proof assistant without adding any axiom to the system. We extract verified OCaml and Haskell programs that can be used by the exchanges and the regulators of the financial markets. We demonstrate the practical applicability of our work by running the verified program on real market data from an exchange to automatically check for violations in the exchange algorithm.
PPoL: A Periodic Channel Hopping Sequence with Nearly Full Rendezvous Diversity
Venue: arXiv cs.DC 2021
Year: 2021

We propose a periodic channel hopping (CH) sequence, called PPoL (Packing the Pencil of Lines in a finite projective plane), for the multichannel rendezvous problem. When $N-1$ is a prime power, its period is $N^2-N+1$, and the number of distinct rendezvous channels of PPoL is at least $N-2$ for any nonzero clock drift. By channel remapping, we construct CH sequences with the maximum time-to-rendezvous (MTTR) bounded by $N^2+3N+3$ if the number of commonly available channels is at least two. This achieves a roughly 50% reduction of the state-of-the-art MTTR bound in the literature.
Sentence Concatenation Approach to Data Augmentation for Neural Machine Translation
Venue: arXiv cs.CL 2021
Year: 2021

Neural machine translation (NMT) has recently gained widespread attention because of its high translation accuracy. However, it shows poor performance in the translation of long sentences, which is a major issue in low-resource languages. It is assumed that this issue is caused by insufficient number of long sentences in the training data. Therefore, this study proposes a simple data augmentation method to handle long sentences. In this method, we use only the given parallel corpora as the training data and generate long sentences by concatenating two sentences. Based on the experimental results, we confirm improvements in long sentence translation by the proposed data augmentation method, despite its simplicity. Moreover, the translation quality is further improved by the proposed method, when combined with back-translation.
Vision Transformer Pruning
Venue: arXiv cs.CV 2021
Year: 2021

Vision transformer has achieved competitive performance on a variety of computer vision applications. However, their storage, run-time memory, and computational demands are hindering the deployment to mobile devices. Here we present a vision transformer pruning approach, which identifies the impacts of dimensions in each layer of transformer and then executes pruning accordingly. By encouraging dimension-wise sparsity in the transformer, important dimensions automatically emerge. A great number of dimensions with small importance scores can be discarded to achieve a high pruning ratio without significantly compromising accuracy. The pipeline for vision transformer pruning is as follows: 1) training with sparsity regularization; 2) pruning dimensions of linear projections; 3) fine-tuning. The reduced parameters and FLOPs ratios of the proposed algorithm are well evaluated and analyzed on ImageNet dataset to demonstrate the effectiveness of our proposed method.
Visually Guided Sound Source Separation and Localization using Self-Supervised Motion Representations
Venue: arXiv cs.CV 2021
Year: 2021

The objective of this paper is to perform audio-visual sound source separation, i.e.~to separate component audios from a mixture based on the videos of sound sources. Moreover, we aim to pinpoint the source location in the input video sequence. Recent works have shown impressive audio-visual separation results when using prior knowledge of the source type (e.g. human playing instrument) and pre-trained motion detectors (e.g. keypoints or optical flows). However, at the same time, the models are limited to a certain application domain. In this paper, we address these limitations and make the following contributions: i) we propose a two-stage architecture, called Appearance and Motion network (AMnet), where the stages specialise to appearance and motion cues, respectively. The entire system is trained in a self-supervised manner; ii) we introduce an Audio-Motion Embedding (AME) framework to explicitly represent the motions that related to sound; iii) we propose an audio-motion transformer architecture for audio and motion feature fusion; iv) we demonstrate state-of-the-art performance on two challenging datasets (MUSIC-21 and AVE) despite the fact that we do not use any pre-trained keypoint detectors or optical flow estimators. Project page: https://ly-zhu.github.io/self-supervised-motion-representations
Quantifying the Need for Attorney Pro Bono Services in Connection with the Social Determinants of Health
Venue: arXiv cs.CY 2021
Year: 2021

The paper estimates the need for additional attorney hours annually to address the legal needs of indigent clients throughout the United States in matters that comprise the so-called social determinants of health (SDoH). The result will inform stakeholders such as policy makers and private donors so they can allocate resources appropriately and design programs to close the do-called justice gap. As a pilot study, the scope of the project covers only a few major justice problems related to the social determinants of health (standard housing, evictions and foreclosures, guardianships for the incapacitated, and victims of domestic violence) because they significantly impact health outcomes and there are data available. Based on our calculations, we estimate that the total number of attorney hours to address only these five legal issues is over 34 million per year. To put that in perspective, the American Bar Association estimated that in 2018, the year from which much of our data comes, the total number of practicing attorneys in the United States was 1,338,678 (Weiss). Thus, to provide the needed hours, every single practicing attorney in the United States would need to contribute about 26 hours a year. While many lawyers do in fact contribute pro bono hours, they address the full range of legal needs that go well beyond just the five we studied.
Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
Venue: arXiv cs.IR 2021
Year: 2021

BERT-based text ranking models have dramatically advanced the state-of-the-art in ad-hoc retrieval, wherein most models tend to consider individual query-document pairs independently. In the mean time, the importance and usefulness to consider the cross-documents interactions and the query-specific characteristics in a ranking model have been repeatedly confirmed, mostly in the context of learning to rank. The BERT-based ranking model, however, has not been able to fully incorporate these two types of ranking context, thereby ignoring the inter-document relationships from the ranking and the differences among queries. To mitigate this gap, in this work, an end-to-end transformer-based ranking model, named Co-BERT, has been proposed to exploit several BERT architectures to calibrate the query-document representations using pseudo relevance feedback before modeling the relevance of a group of documents jointly. Extensive experiments on two standard test collections confirm the effectiveness of the proposed model in improving the performance of text re-ranking over strong fine-tuned BERT-Base baselines. We plan to make our implementation open source to enable further comparisons.
Multi-Perspective Abstractive Answer Summarization
Venue: arXiv cs.CL 2021
Year: 2021

Community Question Answering (CQA) forums such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of questions. Each question thread can receive a large number of answers with different perspectives. The goal of multi-perspective answer summarization is to produce a summary that includes all perspectives of the answer. A major obstacle for multi-perspective, abstractive answer summarization is the absence of a dataset to provide supervision for producing such summaries. This work introduces a novel dataset creation method to automatically create multi-perspective, bullet-point abstractive summaries from an existing CQA forum. Supervision provided by this dataset trains models to inherently produce multi-perspective summaries. Additionally, to train models to output more diverse, faithful answer summaries while retaining multiple perspectives, we propose a multi-reward optimization technique coupled with a sentence-relevance prediction multi-task loss. Our methods demonstrate improved coverage of perspectives and faithfulness as measured by automatic and human evaluations compared to a strong baseline.
Potential Anchoring for imbalanced data classification
Venue: arXiv cs.LG 2021
Year: 2021

Data imbalance remains one of the factors negatively affecting the performance of contemporary machine learning algorithms. One of the most common approaches to reducing the negative impact of data imbalance is preprocessing the original dataset with data-level strategies. In this paper we propose a unified framework for imbalanced data over- and undersampling. The proposed approach utilizes radial basis functions to preserve the original shape of the underlying class distributions during the resampling process. This is done by optimizing the positions of generated synthetic observations with respect to the potential resemblance loss. The final Potential Anchoring algorithm combines over- and undersampling within the proposed framework. The results of the experiments conducted on 60 imbalanced datasets show outperformance of Potential Anchoring over state-of-the-art resampling algorithms, including previously proposed methods that utilize radial basis functions to model class potential. Furthermore, the results of the analysis based on the proposed data complexity index show that Potential Anchoring is particularly well suited for handling naturally complex (i.e. not affected by the presence of noise) datasets.
VSpSR: Explorable Super-Resolution via Variational Sparse Representation
Venue: arXiv cs.CV 2021
Year: 2021

Super-resolution (SR) is an ill-posed problem, which means that infinitely many high-resolution (HR) images can be degraded to the same low-resolution (LR) image. To study the one-to-many stochastic SR mapping, we implicitly represent the non-local self-similarity of natural images and develop a Variational Sparse framework for Super-Resolution (VSpSR) via neural networks. Since every small patch of a HR image can be well approximated by the sparse representation of atoms in an over-complete dictionary, we design a two-branch module, i.e., VSpM, to explore the SR space. Concretely, one branch of VSpM extracts patch-level basis from the LR input, and the other branch infers pixel-wise variational distributions with respect to the sparse coefficients. By repeatedly sampling coefficients, we could obtain infinite sparse representations, and thus generate diverse HR images. According to the preliminary results of NTIRE 2021 challenge on learning SR space, our team (FudanZmic21) ranks 7-th in terms of released scores. The implementation of VSpSR is released at https://zmiclab.github.io/.
Deep Learning in Beyond 5G Networks with Image-based Time-Series Representation
Venue: arXiv cs.NI 2021
Year: 2021

Towards the network innovation, the Beyond Five-Generation (B5G) networks envision the use of machine learning (ML) methods to predict the network conditions and performance indicators in order to best make decisions and allocate resources. In this paper, we propose a new ML approach to accomplish predictions in B5G networks. Instead of handling the time-series in the network domain of values, we transform them into image thus allowing to apply advanced ML methods of Computer Vision field to reach better predictions in B5G networks. Particularly, we analyze different techniques to transform time-series of network measures into image representation, e.g., Recurrence Plots, Markov Transition Fields, and Gramian Angular Fields. Then, we apply deep neural networks with convolutional layers to predict different 5G radio signal quality indicators. When comparing with other ML-based solutions, experimental results from 5G transmission datasets showed the feasibility and small prediction errors of the proposed approach.
Age Range Estimation using MTCNN and VGG-Face Model
Venue: arXiv cs.CV 2021
Year: 2021

The Convolutional Neural Network has amazed us with its usage on several applications. Age range estimation using CNN is emerging due to its application in myriad of areas which makes it a state-of-the-art area for research and improve the estimation accuracy. A deep CNN model is used for identification of people's age range in our proposed work. At first, we extracted only face images from image dataset using MTCNN to remove unnecessary features other than face from the image. Secondly, we used random crop technique for data augmentation to improve the model performance. We have used the concept of transfer learning in our research. A pretrained face recognition model i.e VGG-Face is used to build our model for identification of age range whose performance is evaluated on Adience Benchmark for confirming the efficacy of our work. The performance in test set outperformed existing state-of-the-art by substantial margins.
New Technology, New Rules for Journalism and a New World of Engagement
Venue: arXiv cs.MM 2021
Year: 2021

The ways in which people learn, communicate and engage in discussion have changed profoundly during the past decade. As Jenkins related in her book, The Convergence Crisis: An Impending Paradigm Shift in Advertising, Millenials do not want to be told the whole story. Rather, they want someone to begin a conversation that will engage others to become participants in the development of that story (2015). Technology now allows that to happen, sometimes with unintended and/or ill consequences, but technology also generates a dynamic potential to create international and interactive discourse aimed at addressing shared global challenges.
Learning Fuzzy Clustering for SPECT/CT Segmentation via Convolutional Neural Networks
Venue: arXiv cs.CV 2021
Year: 2021

Quantitative bone single-photon emission computed tomography (QBSPECT) has the potential to provide a better quantitative assessment of bone metastasis than planar bone scintigraphy due to its ability to better quantify activity in overlapping structures. An important element of assessing response of bone metastasis is accurate image segmentation. However, limited by the properties of QBSPECT images, the segmentation of anatomical regions-of-interests (ROIs) still relies heavily on the manual delineation by experts. This work proposes a fast and robust automated segmentation method for partitioning a QBSPECT image into lesion, bone, and background. We present a new unsupervised segmentation loss function and its semi- and supervised variants for training a convolutional neural network (ConvNet). The loss functions were developed based on the objective function of the classical Fuzzy C-means (FCM) algorithm. We conducted a comprehensive study to compare our proposed methods with ConvNets trained using supervised loss functions and conventional clustering methods. The Dice similarity coefficient (DSC) and several other metrics were used as figures of merit as applied to the task of delineating lesion and bone in both simulated and clinical SPECT/CT images. We experimentally demonstrated that the proposed methods yielded good segmentation results on a clinical dataset even though the training was done using realistic simulated images. A ConvNet-based image segmentation method that uses novel loss functions was developed and evaluated. The method can operate in unsupervised, semi-supervised, or fully-supervised modes depending on the availability of annotated training data. The results demonstrated that the proposed method provides fast and robust lesion and bone segmentation for QBSPECT/CT. The method can potentially be applied to other medical image segmentation applications.