Reviewerly/exHarmony huggingface dataset:
Single-stage object detector with local binary pattern for remote sensing images

ABSTRACTThe objects in remote sensing images usually have complex background and appear anywhere in any direction, which poses challenges for deep learning-based object detection. In order to solve the above problems, we embed the local binary pattern (LBP) into deep learning-based object detection to facilitate performance improvements. Specifically, we implement thep LBP via depthwise separable convolution without any learnable parameters, which makes LBP highlight the object-related regions of feature map instead of original images. The convolutional implementation of LBP makes it easy to embed it in deep learning-based object detection, so that deep learning-based object detection can possess the ability of LBP to extract the texture information from complex background without having to learn. The gradients of the branch where LBP convolution is located are blocked and cannot be backpropagated according to proposed LBP convolution, so we propose an effective residual block, called LBPRes (Residual block with Local Binary Pattern) for normal training. To deal with the problem of object direction changes, we introduce the rotation invariance into the LBPRes, denoted as RILBPRes (LBPRes with Rotation Invariance), which makes deep learning-based object detection have the local rotation invariance to cope with the object direction changes. Finally, we construct a single-stage object detection network S2LBPNet (Single Stage object detection Network with Local Binary Pattern) with a local binary pattern and conduct related experiments on the DIOR and HRRSD datasets. According to the experimental results, with a small number of parameters added, S2LBPNet outperforms the YOLOv5s baseline by 2.2% and 5.1% on DIOR and HRRSD datasets, respectively, which prove the superiority of the proposed S2LBPNet.KEYWORDS: Local Binary Patternobject detectionrotation invarianceoptical remote sensing AcknowledgementsThe author would like to thank the DIOR and HRRSD remote sensing image datasets for open accessDisclosure statementNo potential conflict of interest was reported by the author(s).Correction StatementThis article has been corrected with minor changes. These changes do not impact the academic content of the article.Additional informationFundingThis work was supported by National Natural Science Foundation of China [No.12003018], the Fundamental Research Funds for the Central Universities [No.XJS191305], and the China Postdoctoral Science Foundation [No.2018M633471].
Suspicious Activities Recognition in Video Sequences Using DarkNet-NasNet Optimal Deep Features

Human Suspicious Activity Recognition (HSAR) is a critical and active research area in computer vision that relies on artificial intelligence reasoning.Significant advances have been made in this field recently due to important applications such as video surveillance.In video surveillance, humans are monitored through video cameras when doing suspicious activities such as kidnapping, fighting, snatching, and a few more.Although numerous techniques have been introduced in the literature for routine human actions (HAR), very few studies are available for HSAR.This study proposes a deep convolutional neural network (CNN) and optimal featuresbased framework for HSAR in video frames.The framework consists of various stages, including preprocessing video frames, fine-tuning deep models (Darknet 19 and Nasnet mobile) using transfer learning, serial-based feature fusion, feature selection via equilibrium feature optimizer, and neural network classifiers for classification.Fine-tuning two models using some hit and trial methods is the first challenge of this work that was later employed for feature extraction.Next, features are fused in a serial approach, and then an improved optimization method is proposed to select the best features.The proposed technique was evaluated on two action datasets, Hybrid-KTH01 and Hybrid-KTH02, and achieved an accuracy of 99.8% and 99.7%, respectively.The proposed method exhibited higher precision compared to existing state-ofthe-art approaches.
Private, Efficient, and Accurate: Protecting Models Trained by Multi-party Learning with Differential Privacy

Secure multi-party computation-based machine learning, referred to as multi-party learning (MPL for short), has become an important technology to utilize data from multiple parties with privacy preservation. While MPL provides rigorous security guarantees for the computation process, the models trained by MPL are still vulnerable to attacks that solely depend on access to the models. Differential privacy could help to defend against such attacks. However, the accuracy loss brought by differential privacy and the huge communication overhead of secure multi-party computation protocols make it highly challenging to balance the 3-way trade-off between privacy, efficiency, and accuracy.In this paper, we are motivated to resolve the above issue by proposing a solution, referred to as PEA (Private, Efficient, Accurate), which consists of a secure differentially private stochastic gradient descent (DPSGD for short) protocol and two optimization methods. First, we propose a secure DPSGD protocol to enforce DPSGD, which is a popular differentially private machine learning algorithm, in secret sharing-based MPL frameworks. Second, to reduce the accuracy loss led by differential privacy noise and the huge communication overhead of MPL, we propose two optimization methods for the training process of MPL: (1) the data-independent feature extraction method, which aims to simplify the trained model structure; (2) the local data-based global model initialization method, which aims to speed up the convergence of the model training. We implement PEA in two open-source MPL frameworks: TF-Encrypted and Queqiao. The experimental results on various datasets demonstrate the efficiency and effectiveness of PEA. E.g. when ϵ = 2, we can train a differentially private classification model with an accuracy of 88% for CIFAR-10 within 7 minutes under the LAN setting. This result significantly outperforms the one from CryptGPU, one state-of-the-art MPL framework: it costs more than 16 hours to train a non-private deep neural network model on CIFAR-10 with the same accuracy.
The Equalization Losses: Gradient-Driven Training for Long-tailed Object Recognition

Long-tail distribution is widely spread in real-world applications. Due to the extremely small ratio of instances, tail categories often show inferior accuracy. In this paper, we find such performance bottleneck is mainly caused by the imbalanced gradients, which can be categorized into two parts: (1) positive part, deriving from the samples of the same category, and (2) negative part, contributed by other categories. Based on comprehensive experiments, it is also observed that the gradient ratio of accumulated positives to negatives is a good indicator to measure how balanced a category is trained. Inspired by this, we come up with a gradient-driven training mechanism to tackle the long-tail problem: re-balancing the positive/negative gradients dynamically according to current accumulative gradients, with a unified goal of achieving balance gradient ratios. Taking advantage of the simple and flexible gradient mechanism, we introduce a new family of gradient-driven loss functions, namely equalization losses. We conduct extensive experiments on a wide spectrum of visual tasks, including two-stage/single-stage long-tailed object detection (LVIS), long-tailed image classification (ImageNet-LT, Places-LT, iNaturalist), and long-tailed semantic segmentation (ADE20 K). Our method consistently outperforms the baseline models, demonstrating the effectiveness and generalization ability of the proposed equalization losses.
Design of Cloud-Based Real-Time Eye-Tracking Monitoring and Storage System

The rapid development of technology has led to the implementation of data-driven systems whose performance heavily relies on the amount and type of data. In the latest decades, in the field of bioengineering data management, among others, eye-tracking data have become one of the most interesting and essential components for many medical, psychological, and engineering research applications. However, despite the large usage of eye-tracking data in many studies and applications, a strong gap is still present in the literature regarding real-time data collection and management, which leads to strong constraints for the reliability and accuracy of on-time results. To address this gap, this study aims to introduce a system that enables the collection, processing, real-time streaming, and storage of eye-tracking data. The system was developed using the Java programming language, WebSocket protocol, and Representational State Transfer (REST), improving the efficiency in transferring and managing eye-tracking data. The results were computed in two test conditions, i.e., local and online scenarios, within a time window of 100 seconds. The experiments conducted for this study were carried out by comparing the time delay between two different scenarios, even if preliminary results showed a significantly improved performance of data management systems in managing real-time data transfer. Overall, this system can significantly benefit the research community by providing real-time data transfer and storing the data, enabling more extensive studies using eye-tracking data.
Multitask Deep Learning‐Based Whole‐Process System for Automatic Diagnosis of Breast Lesions and Axillary Lymph Node Metastasis Discrimination from Dynamic Contrast‐Enhanced‐<scp>MRI</scp>: A Multicenter Study

Background Accurate diagnosis of breast lesions and discrimination of axillary lymph node (ALN) metastases largely depend on radiologist experience. Purpose To develop a deep learning‐based whole‐process system (DLWPS) for segmentation and diagnosis of breast lesions and discrimination of ALN metastasis. Study Type Retrospective. Population 1760 breast patients, who were divided into training and validation sets (1110 patients), internal (476 patients), and external (174 patients) test sets. Field Strength/Sequence 3.0T/dynamic contrast‐enhanced ( DCE )‐ MRI sequence. Assessment DLWPS was developed using segmentation and classification models. The DLWPS‐based segmentation model was developed by the U‐Net framework, which combined the attention module and the edge feature extraction module. The average score of the output scores of three networks was used as the result of the DLWPS‐based classification model. Moreover, the radiologists' diagnosis without and with the DLWPS‐assistance was explored. To reveal the underlying biological basis of DLWPS, genetic analysis was performed based on RNA‐sequencing data. Statistical Tests Dice similarity coefficient (DI), area under receiver operating characteristic curve (AUC), accuracy, sensitivity, specificity, and kappa value. Results The segmentation model reached a DI of 0.828 and 0.813 in the internal and external test sets, respectively. Within the breast lesions diagnosis, the DLWPS achieved AUCs of 0.973 in internal test set and 0.936 in external test set. For ALN metastasis discrimination, the DLWPS achieved AUCs of 0.927 in internal test set and 0.917 in external test set. The agreement of radiologists improved with the DLWPS‐assistance from 0.547 to 0.794, and from 0.848 to 0.892 in breast lesions diagnosis and ALN metastasis discrimination, respectively. Additionally, 10 breast cancers with ALN metastasis were associated with pathways of aerobic electron transport chain and cytoplasmic translation. Data Conclusion The performance of DLWPS indicates that it can promote radiologists in the judgment of breast lesions and ALN metastasis and nonmetastasis. Level of Evidence 4 Technical Efficacy Stage 3
ExBEHRT: Extended Transformer for Electronic Health Records

In this study, we introduce ExBEHRT, an extended version of BEHRT (BERT applied to electronic health record data) and applied various algorithms to interpret its results. While BEHRT only considers diagnoses and patient age, we extend the feature space to several multi-modal records, namely demographics, clinical characteristics, vital signs, smoking status, diagnoses, procedures, medications and lab tests by applying a novel method to unify the frequencies and temporal dimensions of the different features. We show that additional features significantly improve model performance for various down-stream tasks in different diseases. To ensure robustness, we interpret the model predictions using an adaption of expected gradients, which has not been applied to transformers with EHR data so far and provides more granular interpretations than previous approaches such as feature and token importances. Furthermore, by clustering the models’ representations of oncology patients, we show that the model has implicit understanding of the disease and is able to classify patients with same cancer type into different risk groups. Given the additional features and interpretability, ExBEHRT can help making informed decisions about disease progressions, diagnoses and risk factors of various diseases.
Applicability of 2D algorithms for 3D characterization in digital rocks physics: an example of a machine learning-based super resolution image generation

Abstract Digital rock physics is based on imaging, segmentation and numerical computations of rock samples. Due to challenges regarding the handling of a large 3-dimensional (3D) sample, 2D algorithms have always been attractive. However, in 2D algorithms, the efficiency of the pore structures in the third direction of the generated 3D sample is always questionable. We used four individually captured µCT-images of a given Berea sandstone with different resolutions (12.922, 9.499, 5.775, and 3.436 µm) to evaluate the super-resolution 3D images generated by multistep Super Resolution Double-U-Net (SRDUN), a 2D algorithm. Results show that unrealistic features form in the third direction due to section-wise reconstruction of 2D images. To overcome this issue, we suggest to generate three 3D samples using SRDUN in different directions and then to use one of two strategies: compute the average sample (reconstruction by averaging) or segment one-directional samples and combine them together (binary combination). We numerically compute rock physical properties (porosity, connected porosity, P- and S-wave velocity, permeability and formation factor) to evaluate these models. Results reveal that compared to one-directional samples, harmonic averaging leads to a sample with more similar properties to the original sample. On the other hand, rock physics trends can be calculated using a binary combination strategy by generating low, medium and high porosity samples. These trends are compatible with the properties obtained from one-directional and averaged samples as long as the scale difference between the input and output images of SRDUN is small enough (less than about 3 in our case). By increasing the scale difference, more dispersed results are obtained.
Learning how to detect: A deep reinforcement learning method for whole-slide melanoma histopathology images

Cutaneous melanoma represents one of the most life-threatening malignancies. Histopathological image analysis serves as a vital tool for early melanoma detection. Deep neural network (DNN) models are frequently employed to aid pathologists in enhancing the efficiency and accuracy of diagnoses. However, due to the paucity of well-annotated, high-resolution, whole-slide histopathology image (WSI) datasets, WSIs are typically fragmented into numerous patches during the model training and testing stages. This process disregards the inherent interconnectedness among patches, potentially impeding the models' performance. Additionally, the presence of excess, non-contributing patches extends processing times and introduces substantial computational burdens. To mitigate these issues, we draw inspiration from the clinical decision-making processes of dermatopathologists to propose an innovative, weakly supervised deep reinforcement learning framework, titled Fast medical decision-making in melanoma histopathology images (FastMDP-RL). This framework expedites model inference by reducing the number of irrelevant patches identified within WSIs. FastMDP-RL integrates two DNN-based agents: the search agent (SeAgent) and the decision agent (DeAgent). The SeAgent initiates actions, steered by the image features observed in the current viewing field at various magnifications. Simultaneously, the DeAgent provides labeling probabilities for each patch. We utilize multi-instance learning (MIL) to construct a teacher-guided model (MILTG), serving a dual purpose: rewarding the SeAgent and guiding the DeAgent. Our evaluations were conducted using two melanoma datasets: the publicly accessible TCIA-CM dataset and the proprietary MELSC dataset. Our experimental findings affirm FastMDP-RL's ability to expedite inference and accurately predict WSIs, even in the absence of pixel-level annotations. Moreover, our research investigates the WSI-based interactive environment, encompassing the design of agents, state and reward functions, and feature extractors suitable for melanoma tissue images. This investigation offers valuable insights and references for researchers engaged in related studies. The code is available at: https://github.com/titizheng/FastMDP-RL.
Embracing imperfection for quantum technologies

Solid-state spin qubits unlock applications in nanoscale quantum sensing and are at the forefront of creating distributed, long-distance entanglement that could enable a quantum internet.
Mapping (Metaphorical) Journeys in and Against the Academy

Building on previous work that examines the role of the academic in producing ‘critical research’ for the academy, this chapter explores the use of hand-drawn ‘maps’ as a postdigital exercise in naming and spatialising researcher positionality. We begin by briefly exploring the nature of academic work and how our participation in it orients us to the technologies of empire (and possibilities for resistance to it). We then attend to how metaphors of ‘orientation’ shape conceptions of labour – metaphors such as navigating academic terrain, being on the tenure track, and progressing along career paths. Next, we describe various exercises we undertook using hand-drawn maps to understand various experiences in the academy (i.e., metaphorical terrain) and we discuss the ways in which these activities shifted our orientations to forms of inquiry, collaboration, and representation. This chapter contributes to current discussions of postdigital research by providing further evidence against uncritical digital determinism and an analog/digital binary. Rather, we highlight how the use of seemingly unchanged analog tools can allow for postdigital sense-making, acknowledging the ways in which our sense-making and subjectivities can be understood as postdigital assemblages in relation with the social and material world.
Model-Distributed Inference in Multi-Source Edge Networks

Distributed inference techniques can be broadly classified into data-distributed and model-distributed schemes. In data-distributed inference (DDI), each worker carries the entire deep neural network (DNN) model, but processes only a subset of the data. However, feeding the data to workers results in high communication costs especially when the data is large. An emerging paradigm is model-distributed inference (MDI), where each worker carries only a subset of DNN layers. In MDI, a source device that has data processes a few layers of DNN and sends the output to a neighboring device. This process ends when all layers are processed in a distributed manner. In this paper, we investigate MDI with multiple sources, i.e., when more than one device has data. We design a multisource MDI (MS-MDI), which optimizes task scheduling decisions across multiple source devices and workers. Experimental results on a real-life testbed of NVIDIA Jetson TX2 edge devices show that MS-MDI improves the inference time significantly as compared to baselines.
State-Aware Adversarial Training for Utterance-Level Dialogue Generation

Dialogue generation is a challenging problem because it not only requires us to model the context in a conversation but also to exploit it to generate a coherent and fluent utterance.This paper, aiming for a specific topic of this field, proposes an adversarial training based framework for utterance-level dialogue generation.Technically, we train an encoder-decoder generator simultaneously with a discriminative classifier that make the utterance approximate to the state-aware inputs.Experiments on Mul-tiWoZ 2.0 and MultiWoZ 2.1 datasets show that our method achieves advanced improvements on both automatic and human evaluations, and on the effectiveness of our framework facing low-resource.We further explore the effect of fine-grained augmentations for downstream dialogue state tracking (DST) tasks.Experimental results demonstrate the high-quality data generated by our proposed framework improves the performance over state-of-the-art models.
XPrompt: Exploring the Extreme of Prompt Tuning

Prompt tuning learns soft prompts to condition the frozen Pre-trained Language Models (PLMs) for performing downstream tasks in a parameter-efficient manner.While prompt tuning has gradually reached the performance level of fine-tuning as the model scale increases, there is still a large performance gap between prompt tuning and fine-tuning for models of moderate and small scales (typically less than 11B parameters).In this paper, we empirically show that the trained prompt tokens can have a negative impact on a downstream task and thus degrade its performance.To bridge the gap, we propose a novel PROMPT tuning model with an eXtremely small scale (XPROMPT) under the regime of lottery tickets hypothesis.Specifically, XPROMPT eliminates the negative prompt tokens at different granularity levels through a hierarchical structured pruning, yielding a more parameter-efficient prompt yet with a competitive performance.Comprehensive experiments are carried out on the SuperGLUE tasks, and the results indicate that XPROMPT is able to close the performance gap at smaller model scales. 1
Semantic-Enhanced Differentiable Search Index Inspired by Learning Strategies

Recently, a new paradigm called Differentiable Search Index (DSI) has been proposed for document retrieval, wherein a sequence-to-sequence model is learned to directly map queries to relevant document identifiers. The key idea behind DSI is to fully parameterize traditional ''index-retrieve'' pipelines within a single neural model, by encoding all documents in the corpus into the model parameters. In essence, DSI needs to resolve two major questions: (1) how to assign an identifier to each document, and (2) how to learn the associations between a document and its identifier. In this work, we propose a Semantic-Enhanced DSI model (SE-DSI) motivated by Learning Strategies in the area of Cognitive Psychology. Our approach advances original DSI in two ways: (1) For the document identifier, we take inspiration from Elaboration Strategies in human learning. Specifically, we assign each document an Elaborative Description based on the query generation technique, which is more meaningful than a string of integers in the original DSI; and (2) For the associations between a document and its identifier, we take inspiration from Rehearsal Strategies in human learning. Specifically, we select fine-grained semantic features from a document as Rehearsal Contents to improve document memorization. Both the offline and online experiments show improved retrieval performance over prevailing baselines.