Reviewerly/exHarmony huggingface dataset:
Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers

Our objective is language-based search of large-scale image and video datasets. For this task, the approach that consists of independently mapping text and vision to a joint embedding space, a.k.a. dual encoders, is attractive as retrieval scales and is efficient for billions of images using approximate nearest neighbour search. An alternative approach of using vision-text transformers with cross-attention gives considerable improvements in accuracy over the joint embeddings, but is often inapplicable in practice for large-scale retrieval given the cost of the cross-attention mechanisms required for each sample at test time. This work combines the best of both worlds. We make the following three contributions. First, we equip transformer-based models with a new fine-grained cross-attention architecture, providing significant improvements in retrieval accuracy whilst preserving scalability. Second, we introduce a generic approach for combining a Fast dual encoder model with our Slow but accurate transformer-based model via distillation and reranking. Finally, we validate our approach on the Flickr30K image dataset where we show an increase in inference speed by several orders of magnitude while having results competitive to the state of the art. We also extend our method to the video domain, improving the state of the art on the VATEX dataset.
Adoption of Blockchain Technology through Digital Twins in the Construction Industry 4.0: A PESTELS Approach

The key challenges of the building, engineering, construction, operations, and mining (BECOM) industries are the lack of trust, inefficiencies, and the fragmentation of the information value chain into vulnerable data silos throughout the lifecycle of projects. This paper aims to develop a novel conceptual model for the implementation of blockchain technology (BCT) for digital twin(s) (DT) in the BECOM industry 4.0 to improve trust, cyber security, efficiencies, information management, information sharing, and sustainability. A PESTELS approach is used to review the literature and identify the key challenges affecting BCT adoption for the BECOM industry 4.0. A review of the technical literature on BCT combined with the findings from PESTELS analysis permitted researchers to identify the key technological factors affecting BCT adoption in the industry. This allowed offering a technological framework—namely, the decentralized digital twin cycle (DDTC)—that leverages BCT to address the key technological factors and to ultimately enhance trust, security, decentralization, efficiency, traceability, and transparency of information throughout projects’ lifecycles. The study also identifies the gaps in the integration of BCT with key technologies of industry 4.0, including the internet of things (IoT), building information modeling (BIM), and DT. The framework offered addresses key technological factors and narrows key gaps around network governance, scalability, decentralization, interoperability, energy efficiency, computational requirements, and BCT integration with IoT, BIM, and DT throughout projects’ lifecycles. The model also considers the regulatory aspect and the environmental aspect, and the circular economy (CE). The theoretical framework provides key technological building blocks for industry practitioners to develop the DDTC concept further and implement it through experimental works. Finally, the paper provides an industry-specific analysis and technological approach facilitating BCT adoption through DT to address the key challenges and improve sustainability for the BECOM industry 4.0.
Avoiding Overfitting: A Survey on Regularization Methods for Convolutional Neural Networks

Several image processing tasks, such as image classification and object detection, have been significantly improved using Convolutional Neural Networks (CNN). Like ResNet and EfficientNet, many architectures have achieved outstanding results in at least one dataset by the time of their creation. A critical factor in training concerns the network’s regularization, which prevents the structure from overfitting. This work analyzes several regularization methods developed in the past few years, showing significant improvements for different CNN models. The works are classified into three main areas: the first one is called “data augmentation,” where all the techniques focus on performing changes in the input data. The second, named “internal changes,” aims to describe procedures to modify the feature maps generated by the neural network or the kernels. The last one, called “label,” concerns transforming the labels of a given input. This work presents two main differences comparing to other available surveys about regularization: (i) the first concerns the papers gathered in the manuscript, which are not older than five years, and (ii) the second distinction is about reproducibility, i.e., all works referred here have their code available in public repositories or they have been directly implemented in some framework, such as TensorFlow or Torch.
Tracking the Orientation and Axes Lengths of an Elliptical Extended Object

Extended object tracking considers the simultaneous estimation of the kinematic state and the shape parameters of a moving object based on a varying number of noisy detections. The main challenge in extended object tracking is the nonlinearity and high dimensionality of the estimation problem. This study presents compact closed-form expressions for a recursive Kalman filter that explicitly estimates the orientation and axes lengths of an extended object based on detections that are scattered over the object surface (according to a Gaussian distribution). Existing approaches are either based on Monte Carlo approximations or do not allow for explicitly maintaining all ellipse parameters. The performance of the novel approach is demonstrated with respect to the state of the art by means of simulations.
A Secure and Efficient Data Integrity Verification Scheme for Cloud-IoT Based on Short Signature

The Internet of Things (IoT) is also known as the Internet of everything.As an important part of the new generation of intelligent information technology, the IoT has attracted the attention both of researchers and engineers all over the world.Considering the limited capacity of smart products, the IoT mainly uses cloud computing to expand computing and storage resources.The massive data collected by the sensor are stored in the cloud storage server, also the cloud vulnerability will directly threaten the security and reliability of the IoT.In order to ensure data integrity and availability in the cloud and IoT storage system, users need to verify the integrity of remote data.However, the existing remote data integrity verification schemes are mostly based on the RSA and BLS signature mechanisms.The RSA-based scheme has too much computational overhead.The BLS signature-based scheme needs to adopt a specific hash function, and the batch signature efficiency in the big data environment is low.Therefore, for the computational overhead and signature efficiency issues of these two signature mechanisms, we propose a scheme of data integrity verification based on a short signature algorithm (ZSS signature), which supports privacy protection and public auditing by introducing a trusted third party (TPA).The computational overhead is effectively reduced by reducing hash function overhead in the signature process.Under the assumption of CDH difficult problem, it can resist adaptive chosen-message attacks.The analysis shows that the scheme has a higher efficiency and safety.
Particle Swarm Optimization Algorithm-Extreme Learning Machine (PSO-ELM) Model for Predicting Resilient Modulus of Stabilized Aggregate Bases

Stabilized base/subbase materials provide more structural support and durability to both flexible and rigid pavements than conventional base/subbase materials. For the design of stabilized base/subbase layers in flexible pavements, good performance in terms of resilient modulus (Mr) under wet-dry cycle conditions is required. This study focuses on the development of a Particle Swarm Optimization-based Extreme Learning Machine (PSO-ELM) to predict the performance of stabilized aggregate bases subjected to wet-dry cycles. Furthermore, the performance of the developed PSO-ELM model was compared with the Particle Swarm Optimization-based Artificial Neural Network (PSO-ANN) and Kernel ELM (KELM). The results showed that the PSO-ELM model significantly yielded higher prediction accuracy in terms of the Root Mean Square Error (RMSE), the Mean Absolute Error (MAE), and the coefficient of determination (r2) compared with the other two investigated models, PSO-ANN and KELM. The PSO-ELM was unique in that the predicted Mr values generally yielded the same distribution and trend as the observed Mr data.
Data Poisoning against Differentially-Private Learners: Attacks and Defenses

Data poisoning attacks aim to manipulate the model produced by a learning algorithm by adversarially modifying the training set. We consider differential privacy as a defensive measure against this type of attack. We show that private learners are resistant to data poisoning attacks when the adversary is only able to poison a small number of items. However, this protection degrades as the adversary is allowed to poison more data. We emprically evaluate this protection by designing attack algorithms targeting objective and output perturbation learners, two standard approaches to differentially-private machine learning. Experiments show that our methods are effective when the attacker is allowed to poison sufficiently many training items.
Clustering with Local Density Peaks-Based Minimum Spanning Tree

Clustering analysis has been widely used in statistics, machine learning, pattern recognition, image processing, and so on. It is a great challenge for most existing clustering algorithms to discover clusters with arbitrary shapes. Clustering algorithms based on Minimum spanning tree (MST) are able to discover clusters with arbitrary shapes, but they are time consuming and susceptible to noise points. In this paper, we employ local density peaks (LDP) to represent the whole data set and define a shared neighbors-based distance between local density peaks to better measure the dissimilarity between objects on manifold data. On the basis of local density peaks and the new distance, we propose a novel MST-based clustering algorithm called LDP-MST. It first uses local density peaks to construct MST and then repeatedly cuts the longest edge until a given number of clusters are found. The experimental results on synthetic data sets and real data sets show that our algorithm is competent with state-of-the-art methods when discovering clusters with complex structures.
Knowledge Aware Conversation Generation with Explainable Reasoning over Augmented Graphs

Two types of knowledge, triples from knowledge graphs and texts from documents, have been studied for knowledge aware opendomain conversation generation, in which graph paths can narrow down vertex candidates for knowledge selection decision, and texts can provide rich information for response generation.Fusion of a knowledge graph and texts might yield mutually reinforcing advantages, but there is less study on that.To address this challenge, we propose a knowledge aware chatting machine with three components, an augmented knowledge graph with both triples and texts, knowledge selector, and knowledge aware response generator.For knowledge selection on the graph, we formulate it as a problem of multi-hop graph reasoning to effectively capture conversation flow, which is more explainable and flexible in comparison with previous work.To fully leverage long text information that differentiates our graph from others, we improve a state of the art reasoning algorithm with machine reading comprehension technology.We demonstrate the effectiveness of our system on two datasets in comparison with state-of-the-art models 1 .
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model

Recent breakthroughs of pretrained language models have shown the effectiveness of self-supervised learning for a wide range of natural language processing (NLP) tasks. In addition to standard syntactic and semantic NLP tasks, pretrained models achieve strong improvements on tasks that involve real-world knowledge, suggesting that large-scale language modeling could be an implicit method to capture knowledge. In this work, we further investigate the extent to which pretrained models such as BERT capture knowledge using a zero-shot fact completion task. Moreover, we propose a simple yet effective weakly supervised pretraining objective, which explicitly forces the model to incorporate knowledge about real-world entities. Models trained with our new objective yield significant improvements on the fact completion task. When applied to downstream tasks, our model consistently outperforms BERT on four entity-related question answering datasets (i.e., WebQuestions, TriviaQA, SearchQA and Quasar-T) with an average 2.7 F1 improvements and a standard fine-grained entity typing dataset (i.e., FIGER) with 5.7 accuracy gains.
Active flow control using machine learning: A brief review

Nowadays the rapidly developing artificial intelligence has become a key solution for problems of diverse disciplines, especially those involving big data. Successes in these areas also attract researchers from the community of fluid mechanics, especially in the field of active flow control (AFC). This article surveys recent successful applications of machine learning in AFC, highlights general ideas, and aims at offering a basic outline for those who are interested in this specific topic. In this short review, we focus on two methodologies, i.e., genetic programming (GP) and deep reinforcement learning (DRL), both having been proven effective, efficient, and robust in certain AFC problems, and outline some future prospects that might shed some light for relevant studies.
DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

Many modern object detectors demonstrate outstanding performances by using the mechanism of looking and thinking twice. In this paper, we explore this mechanism in the backbone design for object detection. At the macro level, we propose Recursive Feature Pyramid, which incorporates extra feedback connections from Feature Pyramid Networks into the bottom-up backbone layers. At the micro level, we propose Switchable Atrous Convolution, which convolves the features with different atrous rates and gathers the results using switch functions. Combining them results in DetectoRS, which significantly improves the performances of object detection. On COCO test-dev, DetectoRS achieves state-of-the-art 55.7% box AP for object detection, 48.5% mask AP for instance segmentation, and 50.0% PQ for panoptic segmentation. The code is made publicly available.
Gradient amplification: An efficient way to train deep neural networks

Improving performance of deep learning models and reducing their training times are ongoing challenges in deep neural networks. There are several approaches proposed to address these challenges, one of which is to increase the depth of the neural networks. Such deeper networks not only increase training times, but also suffer from vanishing gradients problem while training. In this work, we propose gradient amplification approach for training deep learning models to prevent vanishing gradients and also develop a training strategy to enable or disable gradient amplification method across several epochs with different learning rates. We perform experiments on VGG-19 and Resnet models (Resnet-18 and Resnet-34), and study the impact of amplification parameters on these models in detail. Our proposed approach improves performance of these deep learning models even at higher learning rates, thereby allowing these models to achieve higher performance with reduced training time.
Towards Debiasing NLU Models from Unknown Biases

NLU models often exploit biases to achieve high dataset-specific performance without properly learning the intended task.Recently proposed debiasing methods are shown to be effective in mitigating this tendency.However, these methods rely on a major assumption that the types of bias should be known a-priori, which limits their application to many NLU tasks and datasets.In this work, we present the first step to bridge this gap by introducing a self-debiasing framework that prevents models from mainly utilizing biases without knowing them in advance.The proposed framework is general and complementary to the existing debiasing methods.We show that it allows these existing methods to retain the improvement on the challenge datasets (i.e., sets of examples designed to expose models' reliance on biases) without specifically targeting certain biases.Furthermore, the evaluation suggests that applying the framework results in improved overall robustness. 1
Quantum network routing and local complementation

Quantum communication between distant parties is based on suitable instances of shared entanglement. For efficiency reasons, in an anticipated quantum network beyond point-to-point communication, it is preferable that many parties can communicate simultaneously over the underlying infrastructure; however, bottlenecks in the network may cause delays. Sharing of multi-partite entangled states between parties offers a solution, allowing for parallel quantum communication. Specifically for the two-pair problem, the butterfly network provides the first instance of such an advantage in a bottleneck scenario. The underlying method differs from standard repeater network approaches in that it uses a graph state instead of maximally entangled pairs to achieve long-distance simultaneous communication. We will demonstrate how graph theoretic tools, and specifically local complementation, help decrease the number of required measurements compared to usual methods applied in repeater schemes. We will examine other examples of network architectures, where deploying local complementation techniques provides an advantage. We will finally consider the problem of extracting graph states for quantum communication via local Clifford operations and Pauli measurements, and discuss that while the general problem is known to be NP-complete, interestingly, for specific classes of structured resources, polynomial time algorithms can be identified.
BLAZE: Blazing Fast Privacy-Preserving Machine Learning

Machine learning tools have illustrated their potential in many significant sectors such as healthcare and finance, to aide in deriving useful inferences. The sensitive and confidential nature of the data, in such sectors, raise natural concerns for the privacy of data. This motivated the area of Privacy-preserving Machine Learning (PPML) where privacy of the data is guaranteed. Typically, ML techniques require large computing power, which leads clients with limited infrastructure to rely on the method of Secure Outsourced Computation (SOC). In SOC setting, the computation is outsourced to a set of specialized and powerful cloud servers and the service is availed on a pay-per-use basis. In this work, we explore PPML techniques in the SOC setting for widely used ML algorithms-- Linear Regression, Logistic Regression, and Neural Networks. We propose BLAZE, a blazing fast PPML framework in the three server setting tolerating one malicious corruption over a ring (\Z{\ell}). BLAZE achieves the stronger security guarantee of fairness (all honest servers get the output whenever the corrupt server obtains the same). Leveraging an input-independent preprocessing phase, BLAZE has a fast input-dependent online phase relying on efficient PPML primitives such as: (i) A dot product protocol for which the communication in the online phase is independent of the vector size, the first of its kind in the three server setting; (ii) A method for truncation that shuns evaluating expensive circuit for Ripple Carry Adders (RCA) and achieves a constant round complexity. This improves over the truncation method of ABY3 (Mohassel et al., CCS 2018) that uses RCA and consumes a round complexity that is of the order of the depth of RCA. An extensive benchmarking of BLAZE for the aforementioned ML algorithms over a 64-bit ring in both WAN and LAN settings shows massive improvements over ABY3.
BSL-1K: Scaling Up Co-articulated Sign Language Recognition Using Mouthing Cues

Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A key stumbling block in making progress towards this goal is a lack of appropriate training data, stemming from the high complexity of sign annotation and a limited supply of qualified annotators. In this work, we introduce a new scalable approach to data collection for sign recognition in continuous videos. We make use of weakly-aligned subtitles for broadcast footage together with a keyword spotting method to automatically localise sign-instances for a vocabulary of 1,000 signs in 1,000 h of video. We make the following contributions: (1) We show how to use mouthing cues from signers to obtain high-quality annotations from video data—the result is the BSL-1K dataset, a collection of British Sign Language (BSL) signs of unprecedented scale; (2) We show that we can use BSL-1K to train strong sign recognition models for co-articulated signs in BSL and that these models additionally form excellent pretraining for other sign languages and benchmarks—we exceed the state of the art on both the MSASL and WLASL benchmarks. Finally, (3) we propose new large-scale evaluation sets for the tasks of sign recognition and sign spotting and provide baselines which we hope will serve to stimulate research in this area.
Multiscale Deep Feature Learning for Human Activity Recognition Using Wearable Sensors

Deep convolutional neural networks (CNNs) achieve state-of-the-art performance in wearable human activity recognition (HAR), which has become a new research trend in ubiquitous computing scenario. Increasing network depth or width can further improve accuracy. However, in order to obtain the optimal HAR performance on mobile platform, it has to consider a reasonable tradeoff between recognition accuracy and resource consumption. Improving the performance of CNNs without increasing memory and computational burden is more beneficial for HAR. In this article, we first propose a new CNN that uses hierarchical-split (HS) idea for a large variety of HAR tasks, which is able to enhance multiscale feature representation ability via capturing a wider range of receptive fields of human activities within one feature layer. Experiments conducted on benchmarks demonstrate that the proposed HS module is an impressive alternative to baseline models with similar model complexity, and can achieve higher recognition performance (e.g., 97.28%, 93.75%, 99.02%, and 79.02% classification accuracies) on UCI-HAR, PAMAP2, WISDM, and UNIMIB-SHAR. Extensive ablation studies are performed to evaluate the effect of the variations of receptive fields on classification performance. Finally, we demonstrate that multiscale receptive fields can help to learn more discriminative features (achieving 94.10% SOTA accuracy) in weakly labeled HAR dataset.