Reviewerly/exHarmony huggingface dataset:
Adaptive Gradient Methods with Dynamic Bound of Learning Rate

Adaptive optimization methods such as AdaGrad, RMSprop and Adam have been proposed to achieve a rapid training process with an element-wise scaling term on learning rates. Though prevailing, they are observed to generalize poorly compared with SGD or even fail to converge due to unstable and extreme learning rates. Recent work has put forward some algorithms such as AMSGrad to tackle this issue but they failed to achieve considerable improvement over existing methods. In our paper, we demonstrate that extreme learning rates can lead to poor performance. We provide new variants of Adam and AMSGrad, called AdaBound and AMSBound respectively, which employ dynamic bounds on learning rates to achieve a gradual and smooth transition from adaptive methods to SGD and give a theoretical proof of convergence. We further conduct experiments on various popular tasks and models, which is often insufficient in previous work. Experimental results show that new variants can eliminate the generalization gap between adaptive methods and SGD and maintain higher learning speed early in training at the same time. Moreover, they can bring significant improvement over their prototypes, especially on complex deep networks. The implementation of the algorithm can be found at this https URL .
Understanding the Difficulty of Training Transformers

Transformers have proved effective in many NLP tasks.However, their training requires non-trivial efforts regarding carefully designing cutting-edge optimizers and learning rate schedulers (e.g., conventional SGD fails to train Transformers effectively).Our objective here is to understand what complicates Transformer training from both empirical and theoretical perspectives.Our analysis reveals that unbalanced gradients are not the root cause of the instability of training.Instead, we identify an amplification effect that influences training substantially-for each layer in a multi-layer Transformer model, heavy dependency on its residual branch makes training unstable, since it amplifies small parameter perturbations (e.g., parameter updates) and results in significant disturbances in the model output.Yet we observe that a light dependency limits the model potential and leads to inferior trained models.Inspired by our analysis, we propose Admin (Adaptive model initialization) to stabilize the early stage's training and unleash its full potential in the late stage.Extensive experiments show that Admin is more stable, converges faster, and leads to better performance 1 .
3D Image Processing using Machine Learning based Input Processing for Man-Machine Interaction

In various real time applications, several assisted services are provided by the human-robot interaction (HRI). The concept of convergence of a three-dimensional (3D) image into a plane-based projection is used for object identification via digital visualization in robotic systems. Recognition errors occur as the projections in various planes are misidentified during the convergence process. These misidentifications in recognition of objects can be reduced by input processing scheme dependent on the projection technique. The conjoining indices are identified by projecting the input image in all possible dimensions and visualizing it. Machine learning algorithm is used for improving the processing speed and accuracy of recognition. Labeled analysis is used for segregation of the intersection without conjoined indices. Errors are prevented by identifying the non-correlating indices in the projections of possible dimension. The inputs are correlated with related inputs that are stored with labels thereby preventing matching of the indices and deviations in the planes. Error, complexity, time and recognition ratio metrics are verified for the proposed model.
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking

Self-supervised pre-training techniques have achieved remarkable progress in Document AI. Most multimodal pre-trained models use a masked language modeling objective to learn bidirectional representations on the text modality, but they differ in pre-training objectives for the image modality. This discrepancy adds difficulty to multimodal representation learning. In this paper, we propose LayoutLMv3 to pre-train multimodal Transformers for Document AI with unified text and image masking. Additionally, LayoutLMv3 is pre-trained with a word-patch alignment objective to learn cross-modal alignment by predicting whether the corresponding image patch of a text word is masked. The simple unified architecture and training objectives make LayoutLMv3 a general-purpose pre-trained model for both text-centric and image-centric Document AI tasks. Experimental results show that LayoutLMv3 achieves state-of-the-art performance not only in text-centric tasks, including form understanding, receipt understanding, and document visual question answering, but also in image-centric tasks such as document image classification and document layout analysis. The code and models are publicly available at https://aka.ms/layoutlmv3.
Hierarchical Attention Network for Image Captioning

Recently, attention mechanism has been successfully applied in image captioning, but the existing attention methods are only established on low-level spatial features or high-level text features, which limits richness of captions. In this paper, we propose a Hierarchical Attention Network (HAN) that enables attention to be calculated on pyramidal hierarchy of features synchronously. The pyramidal hierarchy consists of features on diverse semantic levels, which allows predicting different words according to different features. On the other hand, due to the different modalities of features, a Multivariate Residual Module (MRM) is proposed to learn the joint representations from features. The MRM is able to model projections and extract relevant relations among different features. Furthermore, we introduce a context gate to balance the contribution of different features. Compared with the existing methods, our approach applies hierarchical features and exploits several multimodal integration strategies, which can significantly improve the performance. The HAN is verified on benchmark MSCOCO dataset, and the experimental results indicate that our model outperforms the state-of-the-art methods, achieving a BLEU1 score of 80.9 and a CIDEr score of 121.7 in the Karpathy’s test split.
Compressive Transformers for Long-Range Sequence Modelling

We present the Compressive Transformer, an attentive sequence model which compresses past memories for long-range sequence learning. We find the Compressive Transformer obtains state-of-the-art language modelling results in the WikiText-103 and Enwik8 benchmarks, achieving 17.1 ppl and 0.97bpc respectively. We also find it can model high-frequency speech effectively and can be used as a memory mechanism for RL, demonstrated on an object matching task. To promote the domain of long-range sequence learning, we propose a new open-vocabulary language modelling benchmark derived from books, PG-19.
Ensemble learning of diffractive optical networks

A plethora of research advances have emerged in the fields of optics and photonics that benefit from harnessing the power of machine learning. Specifically, there has been a revival of interest in optical computing hardware, due to its potential advantages for machine learning tasks in terms of parallelization, power efficiency and computation speed. Diffractive Deep Neural Networks (D2NNs) form such an optical computing framework, which benefits from deep learning-based design of successive diffractive layers to all-optically process information as the input light diffracts through these passive layers. D2NNs have demonstrated success in various tasks, including e.g., object classification, spectral-encoding of information, optical pulse shaping and imaging, among others. Here, we significantly improve the inference performance of diffractive optical networks using feature engineering and ensemble learning. After independently training a total of 1252 D2NNs that were diversely engineered with a variety of passive input filters, we applied a pruning algorithm to select an optimized ensemble of D2NNs that collectively improve their image classification accuracy. Through this pruning, we numerically demonstrated that ensembles of N=14 and N=30 D2NNs achieve blind testing accuracies of 61.14% and 62.13%, respectively, on the classification of CIFAR-10 test images, providing an inference improvement of >16% compared to the average performance of the individual D2NNs within each ensemble. These results constitute the highest inference accuracies achieved to date by any diffractive optical neural network design on the same dataset and might provide a significant leapfrog to extend the application space of diffractive optical image classification and machine vision systems.
Memory-Augmented Dense Predictive Coding for Video Representation Learning

The objective of this paper is self-supervised learning from video, in particular for representations for action recognition. We make the following contributions: (i) We propose a new architecture and learning framework Memory-augmented Dense Predictive Coding (MemDPC) for the task. It is trained with a predictive attention mechanism over the set of compressed memories, such that any future states can always be constructed by a convex combination of the condensed representations, allowing to make multiple hypotheses efficiently. (ii) We investigate visual-only self-supervised video representation learning from RGB frames, or from unsupervised optical flow, or both. (iii) We thoroughly evaluate the quality of the learnt representation on four different downstream tasks: action recognition, video retrieval, learning with scarce annotations, and unintentional action classification. In all cases, we demonstrate state-of-the-art or comparable performance over other approaches with orders of magnitude fewer training data.
Salient Object Detection via Integrity Learning

Although current salient object detection (SOD) works have achieved significant progress, they are limited when it comes to the integrity of the predicted salient regions. We define the concept of integrity at both a micro and macro level. Specifically, at the micro level, the model should highlight all parts that belong to a certain salient object. Meanwhile, at the macro level, the model needs to discover all salient objects in a given image. To facilitate integrity learning for SOD, we design a novel Integrity Cognition Network (ICON), which explores three important components for learning strong integrity features. 1) Unlike existing models, which focus more on feature discriminability, we introduce a diverse feature aggregation (DFA) component to aggregate features with various receptive fields (i.e., kernel shape and context) and increase feature diversity. Such diversity is the foundation for mining the integral salient objects. 2) Based on the DFA features, we introduce an integrity channel enhancement (ICE) component with the goal of enhancing feature channels that highlight the integral salient objects, while suppressing the other distracting ones. 3) After extracting the enhanced features, the part-whole verification (PWV) method is employed to determine whether the part and whole object features have strong agreement. Such part-whole agreements can further improve the micro-level integrity for each salient object. To demonstrate the effectiveness of our ICON, comprehensive experiments are conducted on seven challenging benchmarks. Our ICON outperforms the baseline methods in terms of a wide range of metrics. Notably, our ICON achieves ~10% relative improvement over the previous best model in terms of average false negative ratio (FNR), on six datasets.
DBA: Distributed Backdoor Attacks against Federated Learning

Backdoor attacks aim to manipulate a subset of training data by injecting adversarial triggers such that machine learning models trained on the tampered dataset will make arbitrarily (targeted) incorrect prediction on the testset with the same trigger embedded. While federated learning (FL) is capable of aggregating information provided by different parties for training a better model, its distributed learning methodology and inherently heterogeneous data distribution across parties may bring new vulnerabilities. In addition to recent centralized backdoor attacks on FL where each party embeds the same global trigger during training, we propose the distributed backdoor attack (DBA) --- a novel threat assessment framework developed by fully exploiting the distributed nature of FL. DBA decomposes a global trigger pattern into separate local patterns and embed them into the training set of different adversarial parties respectively. Compared to standard centralized backdoors, we show that DBA is substantially more persistent and stealthy against FL on diverse datasets such as finance and image data. We conduct extensive experiments to show that the attack success rate of DBA is significantly higher than centralized backdoors under different settings. Moreover, we find that distributed attacks are indeed more insidious, as DBA can evade two state-of-the-art robust FL algorithms against centralized backdoors. We also provide explanations for the effectiveness of DBA via feature visual interpretation and feature importance ranking. To further explore the properties of DBA, we test the attack performance by varying different trigger factors, including local trigger variations (size, gap, and location), scaling factor in FL, data distribution, and poison ratio and interval. Our proposed DBA and thorough evaluation results shed lights on characterizing the robustness of FL.
Crossbar-Net: A Novel Convolutional Neural Network for Kidney Tumor Segmentation in CT Images

Due to the unpredictable location, fuzzy texture and diverse shape, accurate segmentation of the kidney tumor in CT images is an important yet challenging task.To this end, we in this paper present a cascaded trainable segmentation model termed as Crossbar-Net.Our method combines two novel schemes: (1) we originally proposed the crossbar patches, which consists of two orthogonal non-squared patches (i.e., the vertical patch and horizontal patch).The crossbar patches are able to capture both the global and local appearance information of the kidney tumors from both the vertical and horizontal directions simultaneously.(2) With the obtained crossbar patches, we iteratively train two sub-models (i.e., horizontal sub-model and vertical sub-model) in a cascaded training manner.During the training, the trained sub-models are encouraged to become more focus on the difficult parts of the tumor automatically (i.e., mis-segmented regions).Specifically, the vertical (horizontal) sub-model is required to help segment the mis-segmented regions for the horizontal (vertical) sub-model.Thus, the two sub-models could complement each other to achieve the self-improvement until convergence.In the experiment, we evaluate our method on a real CT kidney tumor dataset which is collected from 94 different patients including 3,500 CT slices.Compared with the state-of-the-art segmentation methods, the results demonstrate the superior performance of our method on the Dice similarity coefficient, true positive fraction, centroid distance and Hausdorff distance.Moreover, to exploit the generalization to other segmentation tasks, we also extend our Crossbar-Net to two related segmentation tasks:(1) cardiac segmentation in MR images and (2) breast mass segmentation in X-ray images, showing the promising results for these two tasks.Our implementation is released at https: //github.com/Qianyu1226/Crossbar-Net.
Transformer-based unsupervised contrastive learning for histopathological image classification

A large-scale and well-annotated dataset is a key factor for the success of deep learning in medical image analysis. However, assembling such large annotations is very challenging, especially for histopathological images with unique characteristics (e.g., gigapixel image size, multiple cancer types, and wide staining variations). To alleviate this issue, self-supervised learning (SSL) could be a promising solution that relies only on unlabeled data to generate informative representations and generalizes well to various downstream tasks even with limited annotations. In this work, we propose a novel SSL strategy called semantically-relevant contrastive learning (SRCL), which compares relevance between instances to mine more positive pairs. Compared to the two views from an instance in traditional contrastive learning, our SRCL aligns multiple positive instances with similar visual concepts, which increases the diversity of positives and then results in more informative representations. We employ a hybrid model (CTransPath) as the backbone, which is designed by integrating a convolutional neural network (CNN) and a multi-scale Swin Transformer architecture. The CTransPath is pretrained on massively unlabeled histopathological images that could serve as a collaborative local-global feature extractor to learn universal feature representations more suitable for tasks in the histopathology image domain. The effectiveness of our SRCL-pretrained CTransPath is investigated on five types of downstream tasks (patch retrieval, patch classification, weakly-supervised whole-slide image classification, mitosis detection, and colorectal adenocarcinoma gland segmentation), covering nine public datasets. The results show that our SRCL-based visual representations not only achieve state-of-the-art performance in each dataset, but are also more robust and transferable than other SSL methods and ImageNet pretraining (both supervised and self-supervised methods). Our code and pretrained model are available at https://github.com/Xiyue-Wang/TransPath.
A Web application for learning and training of mouse handling as an interaction device in digital environments

In this article, we present a solution for training the mouse input device handling, in order to improve the users' digital skills. Knowing that the widespread access to the computer and the Internet is still made by the use of traditional devices, such as, the keyboard and mouse device (and these devices does not provide natural interaction between people and technology), it is necessary to train for the correct handling. Specifically, mouse usage requires movement control, hand posture and finger positioning, as well as, precision clicks for different functions (one click, two clicks, drag and click.) This training is often done in real interaction situations, that could led to inefficiency of use the digital environments and also user's frustration and discouragement. Thus, using simple and intuitive interfaces and providing a set of activities, we allow users to learn and train the motor coordination and basic movements with the mouse. The main results of the study showed that users had a pleasant experience of interaction, considered the interfaces intuitive and affirmed their availability to use it as a training tool.
Innovative Media Projects: New Forms Of Informational Perception (Series “1968.Digital”)

The article considers the peculiarities of the innovative media project “1968Digital. The year when it all started”, included in the short-list of the award “Made in Russia – 2018” as a source of extra intellectual, social and psychological load for the contemporary consumer of information. In conditions of media space transformation with individual mobile device being the main channel of communication, the contemporary net stories are analyzed as the symbolic circuit system. The first series for mobile phones became the answer to technological, communicative, psychological challenges of the 21st century. A special attention is paid to the development of the historic topic with the help of the latest multimedia and transmedia technologies, to the change of the dialogue principles between the addresser and the addressee within the situation of the reference to the past. Another important aspect is the study of specific functioning and perception of the polycode media text demanding special background knowledge and skills of informational consumption. The study of the audience of the project “1968Digital. The year when it all started” showed the following: Internet contacts are often formalized with the appearance of the minimal subject “autonomy” and maximal certain inclusiveness in the ready media project; the authors analyze the innovative media project from the point of view of transformation of perception of contemporary media texts and state the results of the social and psychological survey of the audience.
Soft sensor development using PLSR based multi-kernel ELM

It takes many efforts to establish accurate soft sensor models because of the increasing complication of processes. For the sake of solving this problem, a novel multi-kernel extreme learning machine based on partial least square regression (PLSR) is proposed. In the proposed method, different kernel functions are used for mapping the space of process data to highly nonlinear space. The partial least square regression is adopted to obtain the relationship between the nonlinear space and output layer. To validate the performance of the proposed model, a case study using the High Density Polyethylene process is executed. Simulation results confirm the performance of the proposed model.
EPISODIC HYDROTHERMAL PROCESSES IN AU-SB-W DEPOSITS OF THE STIBNITE DISTRICT, IDAHO, USA: THE VALUE OF IN-SITU TECHNIQUES

The Stibnite mining district in central Idaho hosts large, structurally controlled antimony and gold (~ 200 metric tons) deposits that Midas Gold has applied for a permit to mine. This historic district also supplied tungsten for the WWII effort. New research and mapping by the Idaho Geological Survey and collaborators (Gillerman, et al., 2014, 2019; Stewart et al., 2016) and colleagues at the US Geological Survey (multiple authors, in preparation) confirm the episodic hydrothermal sequence noted by early workers. Early disseminated gold is associated with muscovite alteration and sulfidation of biotite in granitic or metasedimentary host rocks. Later Au, or W and Sb veins and breccias are more fracture-controlled. Radioisotopic ages document a complex sequence of ore deposition from early Paleocene through mid-Eocene time. Disseminated gold, hosted in arsenian pyrite and arsenopyrite, was earliest, followed by carbonate-quartz-pyrite veins with potassium feldspar alteration. Scheelite and stibnite deposition followed the gold. Epithermal quartz-carbonate-pyrite Au-Ag veins and breccias with potassium feldspar envelopes constitute a younger system, possibly related to the onset of Eocene Challis magmatism. Investigation of the hydrothermal systems at Stibnite has required both old and new tools. Petrographic observations, documented photographically from microscope work, distinguish multiple hydrothermal generations and overprinting. Imaging and in-situ microanalyses of zoned pyrite for Au and trace elements, and measurements of common Pb isotope ratios on mineral separates by standard ID-TIMS plus in-situ LA-ICPMS analysis reveal characteristics of pre-ore and ore-bearing sulfides. Arsenic is the major trace element associated with gold-bearing pyrite, but trace copper and antimony are present. Lead isotopes are highly radiogenic and distinct from juvenile signatures of the Eocene magmatism nearby. The in-situ Pb isotope work, conducted on polished thin sections, shows subtle differences between the early gold and later epithermal system. Modelling of the isotopic evolution suggests hydrothermal fluids interacted with Proterozoic and Archean blocks under central Idaho. Utilization of multiple tools has enabled understanding of the hydrothermal history at Stibnite.