Reviewerly/exHarmony huggingface dataset:
A Framework to Model and Control the State of Presence in Virtual Reality Systems

This work is concerned with the modelling and control of the state of presence of a user in a virtual reality system (VRS). The manner how the state of a user interacting with the VRS is measured by some of its physiological signals is discussed. Based on the user-VRS interaction, a novel feedback control schema to manipulate the user state is proposed. In this, the error between the required and the actual user behavior is used by a controller to select the appropriate stimulus that must be executed by the VRS core scenario to reduce the error. The schema includes the modelling of the presence state by a discrete event strategy based on Petri Nets. Based on the obtained model, the control strategy to select the appropriate stimuli is devised. Finally, some real experiments with a VRS prototype are carried out to show the effectiveness of the proposal.
FedMP: Federated Learning through Adaptive Model Pruning in Heterogeneous Edge Computing

Federated learning (FL) has been widely adopted to train machine learning models over massive distributed data sources in edge computing. However, the existing FL frameworks usually suffer from the difficulties of resource limitation and edge heterogeneity. Herein, we design and implement FedMP, an efficient FL framework through adaptive model pruning. We theoretically analyze the impact of pruning ratio on model training performance, and propose to employ a Multi-Armed Bandit based online learning algorithm to adaptively determine different pruning ratios for heterogeneous edge nodes, even without any prior knowledge of their computation and communication capabilities. With adaptive model pruning, FedMP can not only reduce resource consumption but also achieve promising accuracy. To prevent the diverse structures of pruned models from affecting the training convergence, we further present a new parameter synchronization scheme, called Residual Recovery Synchronous Parallel (R2SP), and provide a theoretical convergence guarantee. Extensive experiments on the classical models and datasets demonstrate that FedMP is effective for different heterogeneous scenarios and data distributions, and can provide up to 4.1× speedup compared to the existing FL methods.
An offline learning co-evolutionary algorithm with problem-specific knowledge

The meta-heuristics is an effective way to solve the complex optimization problems. However, the applicability of meta-heuristic is restricted in real applications due to the various characteristics of the corresponding problems. An offline learning co-evolutionary algorithm (OLCA) based on the fitness landscape analysis that introduces the Gaussian estimation of distribution algorithm (EDA) and a variant of differential evolution (DE) for enhancing the search ability, is proposed for complex continuous real-valued problems. The relationship between strategies and fitness landscapes is established by using offline learning of a random forest. The suitable strategy is determined based on the properties of the fitness landscape trained by a random forest before the beginning of the evolutionary process. The proposed OLCA is tested by using the CEC 2017 benchmark test suite and is compared with several state-of-the-art algorithms. The results show that the proposed OLCA is efficient and competitive for solving complex continuous optimization problems. In addition, the effectiveness of the proposed OLCA is also verified by using 19 IEEE CEC 2011 benchmark problems for tackling real-world problems.
Visuals to Text: A Comprehensive Review on Automatic Image Captioning

Image captioning refers to automatic generation of descriptive texts according to the visual content of images. It is a technique integrating multiple disciplines including the computer vision (CV), natural language processing (NLP) and artificial intelligence. In recent years, substantial research efforts have been devoted to generate image caption with impressive progress. To summarize the recent advances in image captioning, we present a comprehensive review on image captioning, covering both traditional methods and recent deep learning-based techniques. Specifically, we first briefly review the early traditional works based on the retrieval and template. Then deep learning-based image captioning researches are focused, which is categorized into the encoder-decoder framework, attention mechanism and training strategies on the basis of model structures and training manners for a detailed introduction. After that, we summarize the publicly available datasets, evaluation metrics and those proposed for specific requirements, and then compare the state of the art methods on the MS COCO dataset. Finally, we provide some discussions on open challenges and future research directions.
Networked Knowledge and Complex Networks: An Engineering View

Along with the development of information technologies such as mobile Internet, information acquisition technology, cloud computing and big data technology, the traditional knowledge engineering and knowledge-based software engineering have undergone fundamental changes where the network plays an increasingly important role. Within this context, it is required to develop new methodologies as well as technical tools for network-based knowledge representation, knowledge services and knowledge engineering. Obviously, the term “network” has different meanings in different scenarios. Meanwhile, some breakthroughs in several bottleneck problems of complex networks promote the developments of the new methodologies and technical tools for network-based knowledge representation, knowledge services and knowledge engineering. This paper first reviews some recent advances on complex networks, and then, in conjunction with knowledge graph, proposes a framework of networked knowledge which models knowledge and its relationships with the perspective of complex networks. For the unique advantages of deep learning in acquiring and processing knowledge, this paper reviews its development and emphasizes the role that it played in the development of knowledge engineering. Finally, some challenges and further trends are discussed.
Path Planning of Mobile Robot based on Improved Ant Colony Algorithm

Aiming at the problems of traditional ant colony optimization algorithm in path planning, such as slow convergence, easy to fall into local optima, and too many inflection points, an improved ant colony algorithm is proposed. In order to reduce the redundant search, the grid map is preprocessed. By improving the composition of the heuristic function and introducing the guiding function, the mobile robot can be guided to find a better path quickly. The algorithm combines ant colony algorithm with genetic algorithm, adopts elite retention strategy, and puts forward the formulas of crossover and mutation probability. The pheromone update mechanism of the algorithm is improved, and the pheromone concentration of different paths is updated differentially, which can speed up the convergence speed of the algorithm. Finally, the optimal path is further optimized, and smooth operations are taken on it. After a large number of comparative simulation experiments, the results show that the improved algorithm is better than other comparative algorithms in complex environment.
Zero-Shot Cross-Modal Retrieval for Remote Sensing Images With Minimal Supervision

The performance of a deep-learning-based model primarily relies on the diversity and size of the training dataset. However, obtaining such a large amount of labeled data for practical remote sensing applications is expensive and labor-intensive. Training protocols have been previously proposed for few-shot learning (FSL) and zero-shot learning (ZSL). However, FSL is not compatible with handling unobserved class data at the inference phase, while ZSL requires many training samples of the seen classes. In this work, we propose a novel training protocol for image retrieval and name it as <i>label-deficit zero-shot learning</i> (LDZSL). We use this novel LDZSL training protocol for the challenging task of cross-sensor data retrieval in remote sensing. This protocol uses very few labeled data samples of the seen classes during training and interprets unobserved class data samples at the inference phase. This strategy is critical as some data modalities are hard to annotate without domain experts. This work proposes a novel bi-level Siamese network to perform the LDZSL cross-sensor retrieval of multispectral and SAR images. We utilize the available geo-referenced SAR and multispectral data to domain align the embedding features of the two modalities. We experimentally demonstrate the proposed model&#x2019;s efficacy using the So2Sat dataset compared to the existing state-of-the-art models of the ZSL framework trained under a reduced training set. We also show the generalizability of the proposed model using a sketch-based image retrieval task. Experimental results on the Earth on Canvas dataset exhibit comparative performance over the literature.
A Hybrid Attention-Based Deep Neural Network for Simultaneous Multi-Sensor Pruning and Human Activity Recognition

With the popularity and development of Internet of Things (IoT) technology, human activity recognition using IoT devices such as wearable sensors can be implemented for various applications. Due to the complexity of activity recognition, multiple homogeneous or heterogeneous sensors are used to obtain excessive information in most wearable activity recognition systems. However, the increased number of sensors and the way of multichannel signal data bring huge challenges to human activity recognition tasks. How to select suitable sensor channels to balance the computational complexity and recognition accuracy has become a major issue. In this article, we extend the sparse group Lasso mechanism to human activity recognition tasks and propose a hybrid attention-based multi-sensor pruning and feature selection deep neural network, called HAP-DNN. This architecture is able to further perform feature selection on the basis of sensor pruning. HAP-DNN consists of three detachable modules: 1) a feature compression & reconstruction module for sensor feature information fusion and restoration; 2) a feature weight calculation module for calculating sensor channel weights and feature weights; and 3) a learning module for classification, which can be regarded as a filter feature selection method. Four public activity recognition data sets are used to verify our proposed architecture, and the experimental results show that HAP-DNN achieves the best classification performance with the least number of retained feature channels.
FPS: Fast Path Planner Algorithm Based on Sparse Visibility Graph and Bidirectional Breadth-First Search

The majority of planning algorithms used are based on the occupancy grid maps, but in complicated situations, the occupancy grid maps have a significant search overhead. This paper proposed a path planner based on the visibility graph (v-graph) for the mobile robot that uses sparse methods to speed up and simplify the construction of the v-graph. Firstly, the complementary grid framework is designed to reduce graph updating iteration costs during the data collection process in each data frame. Secondly, a filter approach based on the edge length and the number of vertices of the obstacle contour is proposed to reduce redundant nodes and edges in the v-graph. Thirdly, a bidirectional breadth-first search is combined into the path searching process in the proposed fast path planner algorithm in order to reduce the waste of exploring space. Finally, the simulation results indicate that the proposed sparse v-graph planner can significantly improve the efficiency of building the v-graph and reduce the time of path search. In highly convoluted unknown or partially known environments, our method is 40% faster than the FAR Planner and produces paths 25% shorter than it. Moreover, the physical experiment shows that the proposed path planner is faster than the FAR Planner in both the v-graph update process and laser process. The method proposed in this paper performs faster when seeking paths than the conventional method based on the occupancy grid.
Exploring preservice teachers’ digital skills, stress, and coping strategies during online lessons amid covid-19 pandemic in Ghana

There are concerns about the association between the level of students' digital skills and the amount of stress they experience. However, evidence from both cross-sectional and longitudinal studies is limited. The main aim of this study was to address this gap in the literature by examining the relationship between preservice teachers' digital skills and stress. The study employed an online cross-sectional survey, which was completed by 661 pre-service teachers enrolled in colleges of education in Ghana. The findings showed that preservice teachers' low digital skills were associated with higher stress levels during the period of online teaching and learning necessitated by the COVID-19 pandemic. The study further revealed that preservice teachers' limited access to resources was a significant cause of stress. Preservice teachers' main coping strategies for reducing stress were positive framing, such as seeking advice from friends and family, seeing something good in what is happening and learning from the experience. Recommendations include providing more training on digital technologies during the initial preparation for teachers.
The elimination of errors caused by shadow in fringe projection profilometry based on deep learning

The fringe projection profilometry (FPP) has been regarded as a classical and mature technique for 3D shape measurement. However, in practical applications, shadow is un-avoided in the imaging process and causes errors in many FPP systems. In this paper, the errors in FPP systems caused by shadow are first analyzed. Then, a direction-aware spatial context module based network is proposed for detecting the shadow regions of a fringe image. Further, a repairment method is developed based on a generative adversarial network combining some simple processes. The training datasets are rendered by a graphic software to easy the training of the networks. The proposed method can repair the shadow regions successfully with only one fringe image and so it can be applied in varieties of FPP systems. The feasibility and the accuracy of improved by the proposed method have been illustrated by abundant experiments.
Multiple Pedestrian Tracking With Graph Attention Map on Urban Road Scene

Pedestrians are often vulnerable users of urban roads and ensuring their safety is a pressing challenge in the filed of intelligent transportation. Multiple pedestrian tracking is one of the key technologies for traffic statistics and abnormal behavior analysis, etc. Detection-based tracking methods have achieved remarkable results and have become mainstream processing schemes. However, target association is still immature and less effective in complex scenarios. In the proposed tracking system, several candidates surrounding each detected pedestrian are selected sparsely, and the associating relationship between the target and these candidates is determined based on a graph attention map. These graph attention maps contain positional correlations of the matching pairs and are applicable with pedestrians’ posture variations. The weighted correlating value is estimated with the positional weighted matrix and merged attention map. The correlating relationship is confirmed with the weighted correlating value and distance matching loss. To enhance the computation efficiency of the graph attention maps for these tracked targets and candidates, feature extraction is processed separately. Convolutional features extracted from one specific middle layer of the backbone network are used to represent each target or candidate. The experiment results show that the proposed tracker achieves better performance than the other five state-of-the-art trackers on three publicly available databases.
AVQBits—Adaptive Video Quality Model Based on Bitstream Information for Various Video Applications

The paper presents <i>AV QBits</i>, a versatile, bitstream-based video quality model. It can be applied in several contexts such as video service monitoring, evaluation of video encoding quality, of gaming video QoE, and even of omnidirectional video quality. In the paper, it is shown that <i>AV QBits</i> predictions closely match video quality ratings obained in various subjective tests with human viewers, for videos up to 4K-UHD resolution (Ultra-High Definition, 3840 x 2180 pixels) and framerates up 120 fps. With the different variants of <i>AV QBits</i> presented in the paper, video quality can be monitored either at the client side, in the network or directly after encoding. The no-reference <i>AV QBits</i> model was developed for different video services and types of input data, reflecting the increasing popularity of Video-on-Demand services and widespread use of HTTP-based adaptive streaming. At its core, <i>AV QBits</i> encompasses the standardized ITU-T P.1204.3 model, with further model instances that can either have restricted or extended input information, depending on the application context. Four different instances of <i>AV QBits</i> are presented, that is, a Mode 3 model with full access to the bitstream, a Mode 0 variant using only metadata such as codec type, framerate, resoution and bitrate as input, a Mode 1 model using Mode 0 information and frame-type and -size information, and a Hybrid Mode 0 model that is based on Mode 0 metadata and the decoded video pixel information. The models are trained on the authors&#x2019; own AVT-PNATS-UHD-1 dataset described in the paper. All models show a highly competitive performance by using AVT-VQDB-UHD-1 as validation dataset, e.g., with the Mode 0 variant yielding a value of 0.890 Pearson Correlation, the Mode 1 model of 0.901, the hybrid no-reference mode 0 model of 0.928 and the model with full bitstream access of 0.942. In addition, all four <i>AV QBits</i> variants are evaluated when applying them out-of-the-box to different media formats such as 360&#x00B0; video, high framerate (HFR) content, or gaming videos. The analysis shows that the ITU-T P.1204.3 and Hybrid Mode 0 instances of <i>AV QBits</i> for the considered use-cases either perform on par with or better than even state-of-the-art full reference, pixel-based models. Furthermore, it is shown that the proposed Mode 0 and Mode 1 variants outperform commonly used no-reference models for the different application scopes. Also, a long-term integration model based on the standardized ITU-T P.1203.3 is presented to estimate ratings of overall audiovisual streaming Quality of Experience (QoE) for sessions of 30 s up to 5 min duration. In the paper, the <i>AV QBits</i> instances with their per-1-sec score output are evaluated as the video quality component of the proposed long-term integration model. All <i>AV QBits</i> variants as well as the long-term integration module are made publicly available for the community for further research.
VFedCS: Optimizing Client Selection for Volatile Federated Learning

Federated learning (FL) has shown great potential as a privacy-preserving solution to training a centralized model based on local data from available clients. However, we argue that, over the course of training, the available clients may exhibit some volatility in terms of the client population, client data, and training status. Considering these volatilities, we propose a new learning scenario termed volatile federated learning (volatile FL) featuring set volatility, statistical volatility, and training volatility. The volatile client set along with the dynamic of clients’ data and the unreliable nature of clients (e.g., unintentional shutdown and network instability) greatly increase the difficulty of client selection. In this article, we formulate and decompose the global problem into two subproblems based on alternating minimization. For an efficient settlement for the proposed selection problem, we quantify the impact of clients’ data and resource heterogeneity for volatile FL and introduce the cumulative effective participation data (CEPD) as an optimization objective. Based on this, we propose upper confidence bound-based greedy selection, dubbed UCB-GS, to address the client selection problem in volatile FL. Theoretically, we prove that the regret of UCB-GS is strictly bounded by a finite constant, justifying its theoretical feasibility. Furthermore, experimental results show that our method significantly reduces the number of training rounds (by up to 62%) while increasing the global model’s accuracy by 7.51%.
Fall Detection System on Smart Walker Based on Multisensor Data Fusion and SPRT Method

Traditional walkers are commonly used for the elderly in social life, which solves the basic problem of walking, but it is difficult to ensure safety when a fall occurs, and the human-computer interaction is poor.The image recognition method or the IMU sensor method fixed on the user, such as a wearable watch, is used by most of the current fall detection methods.Wearable sensors require the user's wearing operation, which is a little troublesome, and the detection accuracy is related to the way of wearing.The image recognition method requires a high-priced camera and a fixed installation position, which is unable to adapt to outdoor activities.We investigated a low-cost method of mounting the sensor on the body of the walker.We propose in this paper an improved fall detection method, namely Precondition and Limit Threshold SPRT (PLT-SPRT), and a novel fall detection system on the smart walker based on PLT-SPRT.The signals of the upper and lower limb sensors are fused based on the Kalman filter algorithm, and the admittance control parameters are obtained through the system identification method.In this study, the improved sequential probability ratio test algorithm is used to set the null hypothesis and the alternative hypothesis, construct the likelihood ratio and optimize the decision function, which is used to judge whether falls occur.The system is simulated by Matlab software, the user intention after fusion is more accurate, and the optimized decision function is judged accurately.Verified by the embedded system based on STM32 of the smart walker equipment in the real world, it can accurately identify the fallen state, with low detection delay, and the fallen state is detected about 160ms earlier than the traditional threshold-based detection algorithm, at the same time, the accuracy is higher than 94.9%, which meets the high real-time requirements of fall detection and is the ideal solution for smart walkers.
Improved Kernel Correlation Filter Based Moving Target Tracking for Robot Grasping

Tracking and grasping moving objects is a hot topic in the field of robots, which provide great potential in the industrial scene and human-computer cooperation. Based on kernel correlation filter and vision 3D reconstruction, this paper proposes a visual-based tracking and grasping method for moving targets. An improved algorithm based on kernel correlation filter is proposed for object tracking. A scale pool is constructed and a scale filter is trained to solve the problem of algorithm scale adaptation. At the same time, the judgment mechanism of tracking results, secondary detection, and modification update mechanisms are added to improve the robustness of the algorithm. Combined with RGB-D camera, the target is reconstructed to obtain the 3D pose the target. The tracking and intercepting strategy is adopted to grasp the moving target. The proposed method is proven to have good performance through data set comparison tests and experiments on real robot systems.