Reviewerly/exHarmony huggingface dataset:
Adaptive Curriculum Learning for Video Captioning

A portion of the data in video captioning datasets are noisy and unsuitable for models to learn at early stages, e.g., there could be a generic 4-word-long caption lacking distinctive details of video content and a 19-word-long description with rare words and complex structure given the average caption length of around 9. The conventional training method, i.e., learning by random sampling indiscriminately from the whole training set, may cause data bias problems and undermine the model performance. In this work, we present a novel learning strategy, Adaptive Curriculum Learning (ACL), to alleviate the adverse effect of such problems. The main idea of our approach is to allow a model to learn from the data within its competence. Specifically, a difficulty measurement is first defined to evaluate the learning difficulty of video-caption pairs, and training data can be ranked accordingly. Then, based on the learning difficulties and model competence, an adaptive sampling approach is developed to provide suitable training subsets for video captioning models in different training stages. Notably, our proposed ACL is applicable to most existing video captioning works as it requires no modifications of the model architecture. Extensive experiments are conducted on mainstream benchmarks, i.e., MSVD and MSR-VTT datasets. The results show that both RNN-based and Transformer-based models achieve consistent performance improvements with our ACL strategy.
A Hybrid Optimization-Based Medical Data Hiding Scheme for Industrial Internet of Things Security

With the development of the industrial internet technology, the medical data exchange in IoT systems has become more prosperous. Specially, more and more medical images produced by industrial and intelligent devices are outsourced to the cloud for convenient use. However, IoT systems deployment poses several medical data security challenges. To address this issue, in this article, a robust medical data hiding scheme based on secure hybrid optimization for industrial scenario image is presented. Specifically, the marked image is obtained through non-subsampled shearlet transform-multiresolution singular value decomposition. In order to generate the dual marks, we employ the Fisher–Yates permutation to produce the scrambled system watermark address for embedding into the mark image. Afterward, the generated mark image is embedded in the chosen coefficients of the cover in an invisible way. After the watermarking, a hybrid optimization-based encryption scheme is utilized to secure the marked image. Extensive experiments demonstrate the invisibility, security, and robustness of our scheme. Further, the superiority of the scheme is elaborated through making the comparison with the other similar algorithms. The solution not only performs the robust exchange of medical data but also protects the privacy of patients.
SWS

Spatial-temporal kernel density visualization (STKDV) has been extensively used in a wide range of applications, e.g., disease outbreak analysis, traffic accident hotspot detection, and crime hotspot detection. While STKDV can provide accurate and comprehensive data visualization, computing STKDV is time-consuming, which is not scalable to large-scale datasets. To address this issue, we develop a new sliding-window-based solution (SWS), which theoretically reduces the time complexity for generating STKDV, without increasing the space complexity. Moreover, we incorporate SWS with the progressive visualization framework, which can continuously output partial visualization results to users (from coarse to fine), until users satisfy the visualization. Our experimental studies on five large-scale datasets show that SWS achieves 1.71x to 24x speedup compared with the state-of-the-art methods.
Historical Information-Guided Class-Incremental Semantic Segmentation in Remote Sensing Images

Despite the extraordinary success of the deep architectures on semantic segmentation for remote sensing (RS) images, they have difficulties in learning new classes from a sequential data stream because of catastrophic forgetting. Continual learning for semantic segmentation (CSS) is an emerging trend for its capability to cope with the above problems effectively. However, old classes from previous steps are collapsed into the background, which further aggravates the challenge of CSS in the RS scene. In this article, we revisit the knowledge distillation (KD) strategy and the characteristics of class-incremental semantic segmentation (CISS) and then present a generalized and effective framework to learn new classes while preserving knowledge of the learned classes. In particular, we propose two novel historical information-guided modules: the feature global perception module and the label reconstruction (LR) module. The former enables the current model to pay more attention to the region related to the old categories identified by the historical information when learning new classes. Meanwhile, the latter retrieves pixels belonging to the learned classes from the background to handle the background shift problem and maintain the high performance of old classes. We have conducted comprehensive experiments on two RS semantic segmentation datasets of Instance Segmentation in Aerial Images Dataset (iSAID) and Gao Fen (GF) challenge semantic segmentation dataset (GCSS). The experimental results outperform the current state-of-the-art methods in most incremental settings, which demonstrates the effectiveness of the proposed framework.
Multi-Agent Embodied Visual Semantic Navigation With Scene Prior Knowledge

In visual semantic navigation, the robot navigates to a target object with egocentric visual observations and the class label of the target is given. It is a meaningful task inspiring a surge of relevant research. However, most of the existing models are only effective for single-agent navigation, and a single agent has low efficiency and poor fault tolerance when conducting more complicated tasks. Multi-agent collaboration can improve the efficiency and has strong application potentials. In this letter, we propose the multi-agent visual semantic navigation, in which multiple agents collaborate with others to find multiple target objects. It is a challenging task that requires agents to learn reasonable collaboration strategies to perform efficient exploration under the restrictions of communication bandwidth. We develop a hierarchical decision framework based on semantic mapping, scene prior knowledge, and communication mechanism to solve this task. The experimental results in unseen scenes with both seen objects and unseen objects illustrate the higher accuracy and efficiency of the proposed model compared with the single-agent model.
Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning

Federated learning (FL), as an emerging edge artificial intelligence paradigm, enables many edge devices to collaboratively train a global model without sharing their private data. To enhance the training efficiency of FL, various algorithms have been proposed, ranging from first-order to second-order methods. However, these algorithms cannot be applied in scenarios where the gradient information is not available, e.g., federated black-box attack and federated hyperparameter tuning. To address this issue, in this paper we propose a derivative-free federated zeroth-order optimization (FedZO) algorithm featured by performing multiple local updates based on stochastic gradient estimators in each communication round and enabling partial device participation. Under non-convex settings, we derive the convergence performance of the FedZO algorithm on non-independent and identically distributed data and characterize the impact of the numbers of local iterates and participating edge devices on the convergence. To enable communication-efficient FedZO over wireless networks, we further propose an over-the-air computation (AirComp) assisted FedZO algorithm. With an appropriate transceiver design, we show that the convergence of AirComp-assisted FedZO can still be preserved under certain signal-to-noise ratio conditions. Simulation results demonstrate the effectiveness of the FedZO algorithm and validate the theoretical observations.
Incorporate opinion-towards for stance detection

Stance detection can help gain different perspectives into important events, e.g., whether people are in favor of or against certain claim. Most previous work use sentiment information to assist in stance detection. However, they do not consider the critical opinion-towards information, i.e. whether the opinions are aimed at target or other objects. In this work, we incorporate opinion-towards information into a multi-task learning model to facilitate our proposed model for better understanding the sentiment information, which effectively improves the performance of stance detection. In particular, we have constructed a novel label relation matrix which constrains two auxiliary tasks in multi-task learning: (1) sentiment classification, and (2) opinion-towards classification. Our extensive experimental results on three publicly available benchmark datasets demonstrate the effectiveness of the proposed model. In addition, we show the importance of opinion-towards information for stance detection through ablation study and visualization analysis.
Medical Image Cryptosystem using Dynamic Josephus Sequence and Chaotic-hash Scrambling

Image encryption is an interesting research topic, especially in medical images. This is due to the importance of security when storing and transmitting digital images. This study proposes an image encryption procedure that combines several methods such as chaotic-hash scrambling to perform the confusion process, combined hash functions, block-based substitution, dynamic bit shifting based on Josephus sequences, and diffusion with logistic maps to obtain strong encryption against various attacks. Based on the test results implemented on medical images, standard images, and special images. The proposed encryption procedure is proven to have resistance to statistical and differential attacks which have been measured by several measuring tools such as histogram analysis, chi-square, entropy, correlation coefficient, NPCR, UACI, avalanche effect, PSNR, and SSIM. In addition, it also has enough key space to withstand brute force attacks.
From Jam Session to Recital: Synchronous Communication and Collaboration Around Data in Organizations

Prior research on communicating with visualization has focused on public presentation and asynchronous individual consumption, such as in the domain of journalism. The visualization research community knows comparatively little about synchronous and multimodal communication around data within organizations, from team meetings to executive briefings. We conducted two qualitative interview studies with individuals who prepare and deliver presentations about data to audiences in organizations. In contrast to prior work, we did not limit our interviews to those who self-identify as data analysts or data scientists. Both studies examined aspects of speaking about data with visual aids such as charts, dashboards, and tables. One study was a retrospective examination of current practices and difficulties, from which we identified three scenarios involving presentations of data. We describe these scenarios using an analogy to musical performance: small collaborative team meetings are akin to jam session, while more structured presentations can range from semi-improvisational performances among peers to formal recitals given to executives or customers. In our second study, we grounded the discussion around three design probes, each examining a different aspect of presenting data: the progressive reveal of visualization to direct attention and advance a narrative, visualization presentation controls that are hidden from the audience's view, and the coordination of a presenter's video with interactive visualization. Our distillation of interviewees' responses surfaced twelve themes, from ways of authoring presentations to creating accessible and engaging audience experiences.
Multi-scale receptive field fusion network for lightweight image super-resolution

In recent years, with the rapid development of deep learning, quantities of convolutional neural network (CNN) based methods are applied to the field of single image super-resolution (SISR), which have excellent performance compared with traditional ones. However, these methods have a large amount of calculation and memory consumption, limiting their application in edge devices. To address this problem, we propose a more lightweight multi-scale receptive field fusion network (MRFN) for fast and accurate SISR. Specifically, we first propose a multi-scale receptive field fusion block to fuse different scales of spatial information and enlarge the overall receptive field simultaneously. Secondly, based on the non-local sparse attention, we introduce a hash-learnable cheap non-local attention to capture long-range dependencies with much fewer parameters and calculations. Moreover, channel attention is also very important for SISR, we further combine directional second-order information and spatial coordinate information into it to enhance its capability, which is called second-order coordinate attention. The cooperation of these components leads to the success of our MRFN. Extensive experimental results show that our method achieves a better trade-off against other state-of-the-art methods in terms of SISR performance and model complexity.
A Wearable Real-Time Character Recognition System Based on Edge Computing-Enabled Deep Learning for Air-Writing

Nowadays, wearable sensors play a vital role in the detection of human motions, innovating an alternate and intuitive form in human–computer interaction (HCI). In this study, we present a novel real-time wearable system for finger air-writing recognition in three-dimensional (3D) space based on the Arduino Nano 33 BLE Sense as an edge device, which can run TensorFlow Lite to realize recognition and classification on the device. This system enables users to have the freedom and flexibility to write characters (10 digits and 26 English lower-case letters) in free space by moving fingers and uses a deep learning algorithm to recognize 36 characters from the motion data captured by inertial measurement units (IMUs) and processed by a microcontroller, which are both embedded in an Arduino Nano 33 BLE Sense. We prepared 63000 air-writing stroke data samples of 35 subjects containing 18 males and 17 females for convolutional neural network (CNN) training and achieved a high recognition accuracy at 97.95%.
Hidden behind the obvious: Misleading keywords and implicitly abusive language on social media

While social media offers freedom of self-expression, abusive language carry significant negative social impact. Driven by the importance of the issue, research in the automated detection of abusive language has witnessed growth and improvement. However, these detection models display a reliance on strongly indicative keywords, such as slurs and profanity. This means that they can falsely (1a) miss abuse without such keywords or (1b) flag non-abuse with such keywords, and that (2) they perform poorly on unseen data. Despite the recognition of these problems, gaps and inconsistencies remain in the literature. In this study, we analyse the impact of keywords from dataset construction to model behaviour in detail, with a focus on how models make mistakes on (1a) and (1b), and how (1a) and (1b) interact with (2). Through the analysis, we provide suggestions for future research to address all three problems.
Unsupervised neural network-enabled spatial-temporal analytics for data authenticity under environmental smart reporting system

Environment, social and governance (ESG) disclosures required on listed companies have aroused considerable interest in both academia and industry for sustainable development and investing. However, the authenticity and credibility of the ESG report exposed to the public remain in doubt due to black box-like reporting processes with massive human involvements. In this study, a framework of environmental smart reporting system (BI-ESRS) based on the blockchain and Internet of Things (IoT) technologies is developed to automate the acquisition of environment-related data and make the reporting reliable and traceable. In addition, we evaluate the authenticity of the data collected from IoT devices, considering human-made counterfeits on measuring instruments for greenwashing. It is anticipated to stimulate companies to submit high-quality data without fake. Specifically, an unsupervised neural network-enabled spatial-temporal analytics (UN-STA) method is devised to achieve anomalies detection and index the data with an authenticity rate. An artificial neural network, self-organizing map (SOM), is applied to construct the prediction model. The received signal strength indicator (RSSI) of Bluetooth low energy (BLE), the vibration amplitude of smart instruments and data uploading interval constitute the input vector for competitive learning. Finally, an experimental simulation is carried out to demonstrate the implementation of the proposed system and method, and their effectiveness has also been testified. Moreover, the sensitivity of the SOM model over the three factors has been analyzed by applying the control variate technique. This work is expected to serve as a reference for practitioners to satisfy similar requirements in the industry and inspire new ideas for scholars.
Masked face recognition: Human versus machine

The recent COVID-19 pandemic has increased the focus on hygienic and contactless identity verification methods. However, the pandemic led to the wide use of face masks, essential to keep the pandemic under control. The effect of wearing a mask on face recognition (FR) in a collaborative environment is a currently sensitive yet understudied issue. Recent reports have tackled this by evaluating the masked probe effect on the performance of automatic FR solutions. However, such solutions can fail in certain processes, leading to the verification task being performed by a human expert. This work provides a joint evaluation and in-depth analyses of the face verification performance of human experts in comparison to state-of-the-art automatic FR solutions. This involves an extensive evaluation by human experts and 4 automatic recognition solutions. The study concludes with a set of take-home messages on different aspects of the correlation between the verification behaviour of humans and machines.
Susceptibility &amp; defense of satellite image-trained convolutional networks to backdoor attacks

With growing opportunities and incentives to disrupt critical space infrastructure systems, cybersecurity of data pipelines associated with satellite images is becoming an increasingly important task. Combined with this threat is a growing body of literature focused on data poisoning and backdoor attacks on deep learning models tailored toward image recognition. To date, there is no work exploring backdoor attacks on deep learning models trained on satellite data. In this study, we evaluate the susceptibility of convolutional networks trained on satellite images to a common trigger-based backdoor attack. We first insert the attack into models trained on two satellite datasets and assess the susceptibility of each model to the attack. We then explore the capability of two state-of-the-art countermeasure techniques based on neuron activity to detect and repair the damage inflicted by the attack. Our results show significant and highly varied attack susceptibility ranging from 0 to 100% with strong dependence on the combination of classes targeted and features common in satellite images. Existing defense strategies are shown to be generally ineffective for this style of attack on satellite data. Finally, we introduce a diagnostic approach for the detection of poisoning vulnerability in satellite image-based deep learning approaches.
Privacy-preserving local analysis of digital trace data: A proof-of-concept

We present PORT, a software platform for local data extraction and analysis of digital trace data. While digital trace data hold huge potential for social-scientific discovery, their most useful parts have been unattainable for scientists because of privacy concerns and prohibitive access to application programming interfaces. Recently, a workflow was introduced allowing citizens to donate their digital traces to scientists. In this workflow, citizens' digital traces are processed locally on their machines before providing informed consent to share a subset of the data with researchers. In this paper, we present the newly developed software PORT that implements the local processing part of this workflow, protecting privacy by shielding sensitive data from outside observers, including the researchers themselves. When using PORT, researchers can tailor the local processing procedure suitable to the data download package and research question. Thus, PORT enables a host of potential applications of social data science to hitherto unobtainable data.
Demystifying the Black Box: The Importance of Interpretability of Predictive Models in Neurocritical Care

Neurocritical care patients are a complex patient population, and to aid clinical decision-making, many models and scoring systems have previously been developed. More recently, techniques from the field of machine learning have been applied to neurocritical care patient data to develop models with high levels of predictive accuracy. However, although these recent models appear clinically promising, their interpretability has often not been considered and they tend to be black box models, making it extremely difficult to understand how the model came to its conclusion. Interpretable machine learning methods have the potential to provide the means to overcome some of these issues but are largely unexplored within the neurocritical care domain. This article examines existing models used in neurocritical care from the perspective of interpretability. Further, the use of interpretable machine learning will be explored, in particular the potential benefits and drawbacks that the techniques may have when applied to neurocritical care data. Finding a solution to the lack of model explanation, transparency, and accountability is important because these issues have the potential to contribute to model trust and clinical acceptance, and, increasingly, regulation is stipulating a right to explanation for decisions made by models and algorithms. To ensure that the prospective gains from sophisticated predictive models to neurocritical care provision can be realized, it is imperative that interpretability of these models is fully considered.
Calibration of line-structured light vision sensor based on free-placed single cylindrical target

This paper proposes a novel calibration method for the Line-structured Light Vision Sensor (L-LVS) based on a straight cylindrical target that can be free placed. The cylindrical target is placed at least once in the measurement space. The camera captures the image of the cylindrical target and the light stripe. Based on the geometric properties of the cylinder and vanishing points, we calculate the 3D centers of the cylindrical end circles and reconstruct the cylindrical quadric surface. Then, multiple 3D points on the cylindrical surface are obtained according to the perspective projection relationship, which finally allows for the reconstruction of the light plane. Both simulation and real experiments show that the proposed method is effective. In the real experiments, the measurement accuracy of a L-LVS based on the proposed calibration method achieves 0.056 mm, when the measuring distance is about 400 mm. Due to the small-size and pose-unconstrainted target, our calibration method is superior to be applied in the confined measurement space in the wild.