OpenRaiser/Intern-Atlas huggingface dataset:
False Negative Reduction in Semantic Segmentation under Domain Shift using Depth Estimation
Venue: arXiv cs.CV 2022
Year: 2022

State-of-the-art deep neural networks demonstrate outstanding performance in semantic segmentation. However, their performance is tied to the domain represented by the training data. Open world scenarios cause inaccurate predictions which is hazardous in safety relevant applications like automated driving. In this work, we enhance semantic segmentation predictions using monocular depth estimation to improve segmentation by reducing the occurrence of non-detected objects in presence of domain shift. To this end, we infer a depth heatmap via a modified segmentation network which generates foreground-background masks, operating in parallel to a given semantic segmentation network. Both segmentation masks are aggregated with a focus on foreground classes (here road users) to reduce false negatives. To also reduce the occurrence of false positives, we apply a pruning based on uncertainty estimates. Our approach is modular in a sense that it post-processes the output of any semantic segmentation network. In our experiments, we observe less non-detected objects of most important classes and an enhanced generalization to other domains compared to the basic semantic segmentation prediction.
A Novel IoT-based Framework for Non-Invasive Human Hygiene Monitoring using Machine Learning Techniques
Venue: arXiv cs.LG 2022
Year: 2022

People's personal hygiene habits speak volumes about the condition of taking care of their bodies and health in daily lifestyle. Maintaining good hygiene practices not only reduces the chances of contracting a disease but could also reduce the risk of spreading illness within the community. Given the current pandemic, daily habits such as washing hands or taking regular showers have taken primary importance among people, especially for the elderly population living alone at home or in an assisted living facility. This paper presents a novel and non-invasive framework for monitoring human hygiene using vibration sensors where we adopt Machine Learning techniques. The approach is based on a combination of a geophone sensor, a digitizer, and a cost-efficient computer board in a practical enclosure. Monitoring daily hygiene routines may help healthcare professionals be proactive rather than reactive in identifying and controlling the spread of potential outbreaks within the community. The experimental result indicates that applying a Support Vector Machine (SVM) for binary classification exhibits a promising accuracy of ~95% in the classification of different hygiene habits. Furthermore, both tree-based classifier (Random Forrest and Decision Tree) outperforms other models by achieving the highest accuracy (100%), which means that classifying hygiene events using vibration and non-invasive sensors is possible for monitoring hygiene activity.
Gradients of O-information: low-order descriptors of high-order dependencies
Venue: arXiv cs.IT 2022
Year: 2022

O-information is an information-theoretic metric that captures the overall balance between redundant and synergistic information shared by groups of three or more variables. To complement the global assessment provided by this metric, here we propose the gradients of the O-information as low-order descriptors that can characterise how high-order effects are localised across a system of interest. We illustrate the capabilities of the proposed framework by revealing the role of specific spins in Ising models with frustration, and on practical data analysis on US macroeconomic data. Our theoretical and empirical analyses demonstrate the potential of these gradients to highlight the contribution of variables in forming high-order informational circuits
Anthropomorphic Twisted String-Actuated Soft Robotic Gripper with Tendon-Based Stiffening
Venue: arXiv cs.RO 2022
Year: 2022

Realizing high-performance soft robotic grippers is challenging because of the inherent limitations of the soft actuators and artificial muscles that drive them, including low force output, small actuation range, and poor compactness. Despite advances in this area, realizing compact soft grippers with high dexterity and force output is still challenging. This paper explores twisted string actuators (TSAs) to drive a soft robotic gripper. TSAs have been used in numerous robotic applications, but their inclusion in soft robots has been limited. The proposed design of the gripper was inspired by the human hand. Tunable stiffness was implemented in the fingers with antagonistic TSAs. The fingers' bending angles, actuation speed, blocked force output, and stiffness tuning were experimentally characterized. The gripper achieved a score of 6 on the Kapandji test and recreated 31 of the 33 grasps of the Feix GRASP taxonomy. It exhibited a maximum grasping force of 72 N, which was almost 13 times its own weight. A comparison study revealed that the proposed gripper exhibited equivalent or superior performance compared to other similar soft grippers.
GaitTAKE: Gait Recognition by Temporal Attention and Keypoint-guided Embedding
Venue: arXiv cs.CV 2022
Year: 2022

Gait recognition, which refers to the recognition or identification of a person based on their body shape and walking styles, derived from video data captured from a distance, is widely used in crime prevention, forensic identification, and social security. However, to the best of our knowledge, most of the existing methods use appearance, posture and temporal feautures without considering a learned temporal attention mechanism for global and local information fusion. In this paper, we propose a novel gait recognition framework, called Temporal Attention and Keypoint-guided Embedding (GaitTAKE), which effectively fuses temporal-attention-based global and local appearance feature and temporal aggregated human pose feature. Experimental results show that our proposed method achieves a new SOTA in gait recognition with rank-1 accuracy of 98.0% (normal), 97.5% (bag) and 92.2% (coat) on the CASIA-B gait dataset; 90.4% accuracy on the OU-MVLP gait dataset.
Production Assessment using a Knowledge Transfer Framework and Evidence Theory
Venue: arXiv cs.HC 2022
Year: 2022

Operational knowledge is one of the most valuable assets in a company, as it provides a strategic advantage over competitors and ensures steady and optimal operation in machines. An (interactive) assessment system on the shop floor can optimize the process and reduce stopovers because it can provide constant valuable information regarding the machine condition to the operators. However, formalizing operational (tacit) knowledge to explicit knowledge is not an easy task. This transformation considers modeling expert knowledge, quantification of knowledge uncertainty, and validation of the acquired knowledge. This study proposes a novel approach for production assessment using a knowledge transfer framework and evidence theory to address the aforementioned challenges. The main contribution of this paper is a methodology for the formalization of tacit knowledge based on an extended failure mode and effect analysis for knowledge extraction, as well as the use of evidence theory for the uncertainty definition of knowledge. Moreover, this approach uses primitive recursive functions for knowledge modeling and proposes a validation strategy of the knowledge using machine data. These elements are integrated into an interactive recommendation system hosted on a backend that uses HoloLens as a visual interface. We demonstrate this approach using an industrial setup: a laboratory bulk good system. The results yield interesting insights, including the knowledge validation, uncertainty behavior of knowledge, and interactive troubleshooting for the machine operator.
Mobile-IRS Assisted Next Generation UAV Communication Networks
Venue: arXiv cs.IT 2022
Year: 2022

Prior research on intelligent reflection surface (IRS)-assisted unmanned aerial vehicle (UAV) communications has focused on a fixed location for the IRS or mounted on a UAV. The assumption that the IRS is located at a fixed position will prohibit mobile users from maximizing many wireless network benefits, such as data rate and coverage. Furthermore, assuming that the IRS is placed on a UAV is impractical for various reasons, including the IRS's weight and size and the speed of wind in severe weather. Unlike previous studies, this study assumes a single UAV and an IRS mounted on a mobile ground vehicle (M-IRS) to be deployed in an Internet-of-Things (IoT) 6G wireless network to maximize the average data rate. Such a methodology for providing wireless coverage using an M-IRS assisted UAV system is expected in smart cities. In this paper, we formulate an optimization problem to find an efficient trajectory for the UAV, an efficient path for the M-IRS, and users' power allocation coefficients that maximize the average data rate for mobile ground users. Due to its intractability, we propose efficient techniques that can help in finding the solution to the optimization problem. First, we show that our dynamic power allocation technique outperforms the fixed power allocation technique in terms of network average sum rate. Then we employ the individual movement model (Random Waypoint Model) in order to represent the users' movements inside the coverage area. Finally, we propose an efficient approach using a Genetic Algorithm (GA) for finding an efficient trajectory for the UAV, and an efficient path for the M-IRS to provide wireless connectivity for mobile users during their movement. We demonstrate through simulations that our methodology can enhance the average data rate by 15\% on average compared with the static IRS and by 25\% on average compared without the IRS system.
Pruning Early Exit Networks
Venue: arXiv cs.LG 2022
Year: 2022

Deep learning models that perform well often have high computational costs. In this paper, we combine two approaches that try to reduce the computational cost while keeping the model performance high: pruning and early exit networks. We evaluate two approaches of pruning early exit networks: (1) pruning the entire network at once, (2) pruning the base network and additional linear classifiers in an ordered fashion. Experimental results show that pruning the entire network at once is a better strategy in general. However, at high accuracy rates, the two approaches have a similar performance, which implies that the processes of pruning and early exit can be separated without loss of optimality.
Convolutional Neural Networks for Time-dependent Classification of Variable-length Time Series
Venue: arXiv cs.LG 2022
Year: 2022

Time series data are often obtained only within a limited time range due to interruptions during observation process. To classify such partial time series, we need to account for 1) the variable-length data drawn from 2) different timestamps. To address the first problem, existing convolutional neural networks use global pooling after convolutional layers to cancel the length differences. This architecture suffers from the trade-off between incorporating entire temporal correlations in long data and avoiding feature collapse for short data. To resolve this tradeoff, we propose Adaptive Multi-scale Pooling, which aggregates features from an adaptive number of layers, i.e., only the first few layers for short data and more layers for long data. Furthermore, to address the second problem, we introduce Temporal Encoding, which embeds the observation timestamps into the intermediate features. Experiments on our private dataset and the UCR/UEA time series archive show that our modules improve classification accuracy especially on short data obtained as partial time series.
Exploring the Effectiveness of Video Perceptual Representation in Blind Video Quality Assessment
Venue: arXiv cs.CV 2022
Year: 2022

With the rapid growth of in-the-wild videos taken by non-specialists, blind video quality assessment (VQA) has become a challenging and demanding problem. Although lots of efforts have been made to solve this problem, it remains unclear how the human visual system (HVS) relates to the temporal quality of videos. Meanwhile, recent work has found that the frames of natural video transformed into the perceptual domain of the HVS tend to form a straight trajectory of the representations. With the obtained insight that distortion impairs the perceived video quality and results in a curved trajectory of the perceptual representation, we propose a temporal perceptual quality index (TPQI) to measure the temporal distortion by describing the graphic morphology of the representation. Specifically, we first extract the video perceptual representations from the lateral geniculate nucleus (LGN) and primary visual area (V1) of the HVS, and then measure the straightness and compactness of their trajectories to quantify the degradation in naturalness and content continuity of video. Experiments show that the perceptual representation in the HVS is an effective way of predicting subjective temporal quality, and thus TPQI can, for the first time, achieve comparable performance to the spatial quality metric and be even more effective in assessing videos with large temporal variations. We further demonstrate that by combining with NIQE, a spatial quality metric, TPQI can achieve top performance over popular in-the-wild video datasets. More importantly, TPQI does not require any additional information beyond the video being evaluated and thus can be applied to any datasets without parameter tuning. Source code is available at https://github.com/UoLMM/TPQI-VQA.
TGRMPT: A Head-Shoulder Aided Multi-Person Tracker and a New Large-Scale Dataset for Tour-Guide Robot
Venue: arXiv cs.CV 2022
Year: 2022

A service robot serving safely and politely needs to track the surrounding people robustly, especially for Tour-Guide Robot (TGR). However, existing multi-object tracking (MOT) or multi-person tracking (MPT) methods are not applicable to TGR for the following reasons: 1. lacking relevant large-scale datasets; 2. lacking applicable metrics to evaluate trackers. In this work, we target the visual perceptual tasks for TGR and present the TGRDB dataset, a novel large-scale multi-person tracking dataset containing roughly 5.6 hours of annotated videos and over 450 long-term trajectories. Besides, we propose a more applicable metric to evaluate trackers using our dataset. As part of our work, we present TGRMPT, a novel MPT system that incorporates information from head shoulder and whole body, and achieves state-of-the-art performance. We have released our codes and dataset in https://github.com/wenwenzju/TGRMPT.
An Infinitary Proof Theory of Linear Logic Ensuring Fair Termination in the Linear $π$-Calculus
Venue: arXiv cs.LO 2022
Year: 2022

Fair termination is the property of programs that may diverge "in principle" but that terminate "in practice", i.e. under suitable fairness assumptions concerning the resolution of non-deterministic choices. We study a conservative extension of $μ$MALL$^\infty$, the infinitary proof system of the multiplicative additive fragment of linear logic with least and greatest fixed points, such that cut elimination corresponds to fair termination. Proof terms are processes of $π$LIN, a variant of the linear $π$-calculus with (co)recursive types into which binary and (some) multiparty sessions can be encoded. As a result we obtain a behavioral type system for $π$LIN (and indirectly for session calculi through their encoding into $π$LIN) that ensures fair termination: although well-typed processes may engage in arbitrarily long interactions, they are fairly guaranteed to eventually perform all pending actions.
A Deep Learning-Based Framework for Low Complexity Multi-User MIMO Precoding Design
Venue: arXiv cs.IT 2022
Year: 2022

Using precoding to suppress multi-user interference is a well-known technique to improve spectra efficiency in multiuser multiple-input multiple-output (MU-MIMO) systems, and the pursuit of high performance and low complexity precoding method has been the focus in the last decade. The traditional algorithms including the zero-forcing (ZF) algorithm and the weighted minimum mean square error (WMMSE) algorithm failed to achieve a satisfactory trade-off between complexity and performance. In this paper, leveraging on the power of deep learning, we propose a low-complexity precoding design framework for MU-MIMO systems. The key idea is to transform the MIMO precoding problem into the multiple-input single-output precoding problem, where the optimal precoding structure can be obtained in closed-form. A customized deep neural network is designed to fit the mapping from the channels to the precoding matrix. In addition, the technique of input dimensionality reduction, network pruning, and recovery module compression are used to further improve the computational efficiency. Furthermore, the extension to the practical MIMO orthogonal frequency-division multiplexing (MIMO-OFDM) system is studied. Simulation results show that the proposed low-complexity precoding scheme achieves similar performance as the WMMSE algorithm with very low computational complexity.
Towards Intrinsic Common Discriminative Features Learning for Face Forgery Detection using Adversarial Learning
Venue: arXiv cs.CV 2022
Year: 2022

Existing face forgery detection methods usually treat face forgery detection as a binary classification problem and adopt deep convolution neural networks to learn discriminative features. The ideal discriminative features should be only related to the real/fake labels of facial images. However, we observe that the features learned by vanilla classification networks are correlated to unnecessary properties, such as forgery methods and facial identities. Such phenomenon would limit forgery detection performance especially for the generalization ability. Motivated by this, we propose a novel method which utilizes adversarial learning to eliminate the negative effect of different forgery methods and facial identities, which helps classification network to learn intrinsic common discriminative features for face forgery detection. To leverage data lacking ground truth label of facial identities, we design a special identity discriminator based on similarity information derived from off-the-shelf face recognition model. With the help of adversarial learning, our face forgery detection model learns to extract common discriminative features through eliminating the effect of forgery methods and facial identities. Extensive experiments demonstrate the effectiveness of the proposed method under both intra-dataset and cross-dataset evaluation settings.
Pick the Right Co-Worker: Online Assessment of Cognitive Ergonomics in Human-Robot Collaborative Assembly
Venue: arXiv cs.RO 2022
Year: 2022

Human-robot collaborative assembly systems enhance the efficiency and productivity of the workplace but may increase the workers' cognitive demand. This paper proposes an online and quantitative framework to assess the cognitive workload induced by the interaction with a co-worker, either a human operator or an industrial collaborative robot with different control strategies. The approach monitors the operator's attention distribution and upper-body kinematics benefiting from the input images of a low-cost stereo camera and cutting-edge artificial intelligence algorithms (i.e. head pose estimation and skeleton tracking). Three experimental scenarios with variations in workstation features and interaction modalities were designed to test the performance of our online method against state-of-the-art offline measurements. Results proved that our vision-based cognitive load assessment has the potential to be integrated into the new generation of collaborative robotic technologies. The latter would enable human cognitive state monitoring and robot control strategy adaptation for improving human comfort, ergonomics, and trust in automation.
Gestural and Touchscreen Interaction for Human-Robot Collaboration: a Comparative Study
Venue: arXiv cs.RO 2022
Year: 2022

Close human-robot interaction (HRI), especially in industrial scenarios, has been vastly investigated for the advantages of combining human and robot skills. For an effective HRI, the validity of currently available human-machine communication media or tools should be questioned, and new communication modalities should be explored. This article proposes a modular architecture allowing human operators to interact with robots through different modalities. In particular, we implemented the architecture to handle gestural and touchscreen input, respectively, using a smartwatch and a tablet. Finally, we performed a comparative user experience study between these two modalities.
FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis
Venue: arXiv cs.SD 2022
Year: 2022

Unconstrained lip-to-speech synthesis aims to generate corresponding speeches from silent videos of talking faces with no restriction on head poses or vocabulary. Current works mainly use sequence-to-sequence models to solve this problem, either in an autoregressive architecture or a flow-based non-autoregressive architecture. However, these models suffer from several drawbacks: 1) Instead of directly generating audios, they use a two-stage pipeline that first generates mel-spectrograms and then reconstructs audios from the spectrograms. This causes cumbersome deployment and degradation of speech quality due to error propagation; 2) The audio reconstruction algorithm used by these models limits the inference speed and audio quality, while neural vocoders are not available for these models since their output spectrograms are not accurate enough; 3) The autoregressive model suffers from high inference latency, while the flow-based model has high memory occupancy: neither of them is efficient enough in both time and memory usage. To tackle these problems, we propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency, and has a relatively small model size. Besides, different from the widely used 3D-CNN visual frontend for lip movement encoding, we for the first time propose a transformer-based visual frontend for this task. Experiments show that our model achieves $19.76\times$ speedup for audio waveform generation compared with the current autoregressive model on input sequences of 3 seconds, and obtains superior audio quality.