Reviewerly/exHarmony huggingface dataset:
An Efficient and Robust Hand Gesture Recognition System of Sign Language Employing Finetuned Inception-V3 and Efficientnet-B0 Network

Hand Gesture Recognition (HGR) is a promising research area with an extensive range of applications, such as surgery, video game techniques, and sign language translation, where sign language is a complicated structured form of hand gestures. The fundamental building blocks of structured expressions in sign language are the arrangement of the fingers, the orientation of the hand, and the hand’s position concerning the body. The importance of HGR has increased due to the increasing number of touchless applications and the rapid growth of the hearing-impaired population. Therefore, real-time HGR is one of the most effective interaction methods between computers and humans. Developing a user-free interface with good recognition performance should be the goal of real-time HGR systems. Nowadays, Convolutional Neural Network (CNN) shows great recognition rates for different image-level classification tasks. It is challenging to train deep CNN networks like VGG-16, VGG-19, Inception-v3, and Efficientnet-B0 from scratch because only some significant labeled image datasets are available for static hand gesture images. However, an efficient and robust hand gesture recognition system of sign language employing finetuned Inception-v3 and Efficientnet-Bo network is proposed to identify hand gestures using a comparative small HGR dataset. Experiments show that Inception-v3 achieved 90% accuracy and 0.93% precision, 0.91% recall, and 0.90% f1-score, respectively, while EfficientNet-B0 achieved 99% accuracy and 0.98%, 0.97%, 0.98%, precision, recall, and f1-score respectively.
Heavy tails and pruning in programmable photonic circuits for universal unitaries

Developing hardware for high-dimensional unitary operators plays a vital role in implementing quantum computations and deep learning accelerations. Programmable photonic circuits are singularly promising candidates for universal unitaries owing to intrinsic unitarity, ultrafast tunability and energy efficiency of photonic platforms. Nonetheless, when the scale of a photonic circuit increases, the effects of noise on the fidelity of quantum operators and deep learning weight matrices become more severe. Here we demonstrate a nontrivial stochastic nature of large-scale programmable photonic circuits-heavy-tailed distributions of rotation operators-that enables the development of high-fidelity universal unitaries through designed pruning of superfluous rotations. The power law and the Pareto principle for the conventional architecture of programmable photonic circuits are revealed with the presence of hub phase shifters, allowing for the application of network pruning to the design of photonic hardware. For the Clements design of programmable photonic circuits, we extract a universal architecture for pruning random unitary matrices and prove that "the bad is sometimes better to be removed" to achieve high fidelity and energy efficiency. This result lowers the hurdle for high fidelity in large-scale quantum computing and photonic deep learning accelerators.
Modelling with AAS and RDF in Industry 4.0

Industry 4.0 has proposed the Asset Administration Shell (AAS) model for digital twins. This model should help to solve interoperability issues, a topic that is also addressed by the Semantic Web and its Resource Description Framework (RDF). AAS and RDF-based models have their own strengths. AAS models are easier to integrate with operational technologies in a production environment, whereas RDF-based models offer more semantic expressiveness and advanced querying. In the Horizon MAS4AI project we found that both modelling paradigms can complement each other to develop agentbased digital twins for modular production environments. In this work we propose two different approaches to bridge both modelling paradigms. First we define a set of mapping rules to generate an AAS model from a given RDF-based model, supporting model development. Secondly, we propose to use RDF-based models to generate a digital shadow of AASs to improve semantic discoverability. Preliminary results demonstrate that heterogeneity of metamodels does not exclude achieving semantic interoperability, as well as that greater functionality can be obtained compared to using both models in isolation. The solutions will be further developed in collaboration with pilot lines in the MAS4AI project.
3D Object Detection and Tracking Based on Lidar-Camera Fusion and IMM-UKF Algorithm Towards Highway Driving

In this work, we propose DTFI: a 3D object <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">D</b> etection and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b> racking approach consisting of lidar-camera <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">F</b> usion-based 3D object detection and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">I</b> nteracting multiple model with unscented Kalman filter (IMM-UKF) based tracking algorithm towards highway driving. For the 3D object detection, an end-to-end learnable architecture fuses the point cloud from the lidar and images from the camera to generate 3D bounding boxes. First, the point cloud is augmented by the full-resolution image-only feature maps in the architecture. In addition, to address the misalignment of raw data due to calibration errors, windowed convolution is used to match multimodal data. Then, new features are extracted from the augmented point cloud for generating bounding boxes. Moreover, to achieve real-time performance in highway scenarios, a fast encoding network is used to extract features. With the object detection results, multiple object tracking (MOT) is performed, which includes object state estimation and data association. The IMM-UKF algorithm is proposed to estimate the 3D space state to deal with the movement uncertainty of the objects, and the Hungarian algorithm is used for data association. Furthermore, to set the appropriate covariance and the state switching matrix for specific scenarios, the Particle Swarm Optimization (PSO) algorithm is used. To evaluate the performance of the proposed DTFI, comparisons with related methods are made on the KITTI testing benchmark. Experimental results show that the proposed DTFI outperforms the existing fusion-based detection and tracking methods at 30 Hz, which can meet the essential requirement of highway scenarios.
Evolution Of Machine Learning Algorithms For Enhancement Of Self-Driving Vehicles Security

In recent years, autonomous vehicles have been a hot topic of debate. Several major automakers, including many worldwide companies, are attempting to be pioneers in autonomous vehicle technology. Google Waymo, and Aptiv, for example, are all working on self-driving car technology. Radar, Lidar, sonar, GPS, and odometer are some of the technologies utilized in the development of autonomous vehicles to recognize their surroundings. An automatic control system is used to control navigation based on the data collected from these sensors. This study will look at how the CNN deep learning algorithm can be used to recognize the surrounding environment and produce the automatic navigation required for self-driving cars. The designed system will generate and learn the data set ahead of time, then use the learning outputs in an open simulation environment. By analysing the settings of an autonomous car, this simulation displays a high level of accuracy in learning to control it. This not only focused on simulation but also focused on predicting a high accuracy model which will be more scalable.
Analyzing Memory Access Traces of Deep Learning Workloads for Efficient Memory Management

Due to the recent advances in artificial intelligence (AI) and high-performance computing technologies, deep learning is actively used in various application domains of the fourth industrial revolution. Since the data size of deep learning increasingly grows, analyzing the memory reference characteristics of AI workloads becomes important. In this article, we analyze the memory access traces of deep learning workloads consisting of text and image data. Specifically, we analyze read and write operations separately for instruction and data memory accesses. Based on our analysis, we find out the following characteristics that are quite different to memory access traces of conventional desktop workloads. First, when comparing instruction and data accesses, instruction access accounts for a little portion of memory accesses in deep learning workloads, which is quite different from traditional workloads. Specifically, instruction access accounts for 1-3.3% in deep learning workloads while 15-30% in traditional workloads. Second, when comparing read and write accesses, write access accounts for 64-80%, which is also different to traditional workloads where write accounts for 6-55%. Third, although write access accounts for the majority of memory accesses, it exhibits the low access skewness. Specifically, the Zipf parameter of write access is about 0.3, and the parameter is smaller in text workloads than image workloads. Fourth, in predicting re-reference likelihood, recency ranking is important in instruction and data read accesses, but frequency ranking is necessary in write accesses for accurate estimation. We expect that the characterization study in this article will guide for managing memory systems of deep learning workloads.
Analyzing the Interaction between Tweet Sentiments and Price Volatility of Cryptocurrencies

With growing interest toward investments in the cryptocurrency market, prediction of the volatility of the price increasingly becomes important. Given the popularity of social media activity to reflect market trends in recent years, sentiment analysis has been recognized as a great contributing factor to predict financial markets. Using a sample of Bitcoin and Ethereum trade data, this study intends to provide insights on the association between twitter activity about cryptocurrencies and fluctuations of their price. To this end, we implement regression analysis alongside Vector Autoregression method to examine to what extent sentiment-related measures are capable of explaining the volatility of the prices of cryptocurrencies and whether the mutual influence of sentiment and volatility improves the accuracy of the model. Results indicate that the accuracy of predictions vary across the two tested cryptocurrencies, and also two different lexicon approaches used to calculate sentiment scores.
Improved Classification of Blurred Images with Deep-Learning Networks Using Lucy-Richardson-Rosen Algorithm

Pattern recognition techniques form the heart of most, if not all, incoherent linear shift-invariant systems. When an object is recorded using a camera, the object information is sampled by the point spread function (PSF) of the system, replacing every object point with the PSF in the sensor. The PSF is a sharp Kronecker Delta-like function when the numerical aperture (NA) is large with no aberrations. When the NA is small, and the system has aberrations, the PSF appears blurred. In the case of aberrations, if the PSF is known, then the blurred object image can be deblurred by scanning the PSF over the recorded object intensity pattern and looking for pattern matching conditions through a mathematical process called correlation. Deep learning-based image classification for computer vision applications gained attention in recent years. The classification probability is highly dependent on the quality of images as even a minor blur can significantly alter the image classification results. In this study, a recently developed deblurring method, the Lucy-Richardson-Rosen algorithm (LR2A), was implemented to computationally refocus images recorded in the presence of spatio-spectral aberrations. The performance of LR2A was compared against the parent techniques: Lucy-Richardson algorithm and non-linear reconstruction. LR2A exhibited a superior deblurring capability even in extreme cases of spatio-spectral aberrations. Experimental results of deblurring a picture recorded using high-resolution smartphone cameras are presented. LR2A was implemented to significantly improve the performances of the widely used deep convolutional neural networks for image classification.
The Internet of Vehicles and Sustainability—Reflections on Environmental, Social, and Corporate Governance

The Internet of Vehicles (IoV) has generated great interest among researchers from different disciplines as it is multidisciplinary research. Sustainability for the IoV requires solutions from different perspectives, particularly in the context of environmental, social, and corporate governance. This review paper examines each of the mentioned perspectives of IoV research which were conducted among at least one of these three perspectives. On the one hand, this allows determining how widely research on the IoV system has been conducted. Moreover, it shows the directions of research on the IoV. On the other hand, it determines whether and how the IoV research is linked to each of the perspectives separately and analyses this link from a global perspective as well; i.e., it analyses the survey data in terms of the data’s relationship to all the perspectives as a group. As one of the research results, a conceptual model of IoV systems allocating the ESG perspectives was developed. The current research has shown that consideration of IoV systems in the context of these three perspectives (treated both individually and collectively) is still limited. A balanced approach towards these IoV systems is still required. Therefore, the paper consists of a survey of the current research related to the sustainability of the IoV from the three mentioned perspectives, aiming to give a balanced view of the importance of the three perspectives for IoV systems.
An automatic improved facial expression recognition for masked faces

Automatic facial expression recognition (AFER), sometimes referred to as emotional recognition, is important for socializing. Automatic methods in the past two years faced challenges due to Covid-19 and the vital wearing of a mask. Machine learning techniques tremendously increase the amount of data processed and achieved good results in such AFER to detect emotions; however, those techniques are not designed for masked faces and thus achieved poor recognition. This paper introduces a hybrid convolutional neural network aided by a local binary pattern to extract features in an accurate way, especially for masked faces. The basic seven emotions classified into anger, happiness, sadness, surprise, contempt, disgust, and fear are to be recognized. The proposed method is applied on two datasets: the first represents CK and CK +, while the second represents M-LFW-FER. Obtained results show that emotion recognition with a face mask achieved an accuracy of 70.76% on three emotions. Results are compared to existing techniques and show significant improvement.
Enhanced Cyber Attack Detection Process for Internet of Health Things (IoHT) Devices Using Deep Neural Network

Internet of Health Things plays a vital role in day-to-day life by providing electronic healthcare services and has the capacity to increase the quality of patient care. Internet of Health Things (IoHT) devices and applications have been growing rapidly in recent years, becoming extensively vulnerable to cyber-attacks since the devices are small and heterogeneous. In addition, it is doubly significant when IoHT involves devices used in healthcare domain. Consequently, it is essential to develop a resilient cyber-attack detection system in the Internet of Health Things environment for mitigating the security risks and preventing Internet of Health Things devices from becoming exposed to cyber-attacks. Artificial intelligence plays a primary role in anomaly detection. In this paper, a deep neural network-based cyber-attack detection system is built by employing artificial intelligence on latest ECU-IoHT dataset to uncover cyber-attacks in Internet of Health Things environment. The proposed deep neural network system achieves average higher performance accuracy of 99.85%, an average area under receiver operator characteristic curve 0.99 and the false positive rate is 0.01. It is evident from the experimental result that the proposed system attains higher detection rate than the existing methods.
Offline Pre-trained Multi-agent Decision Transformer

Abstract Offline reinforcement learning leverages previously collected offline datasets to learn optimal policies with no necessity to access the real environment. Such a paradigm is also desirable for multi-agent reinforcement learning (MARL) tasks, given the combinatorially increased interactions among agents and with the environment. However, in MARL, the paradigm of offline pre-training with online fine-tuning has not been studied, nor even datasets or benchmarks for offline MARL research are available. In this paper, we facilitate the research by providing large-scale datasets and using them to examine the usage of the decision transformer in the context of MARL. We investigate the generalization of MARL offline pre-training in the following three aspects: 1) between single agents and multiple agents, 2) from offline pretraining to online fine tuning, and 3) to that of multiple downstream tasks with few-shot and zero-shot capabilities. We start by introducing the first offline MARL dataset with diverse quality levels based on the StarCraftII environment, and then propose the novel architecture of multi-agent decision transformer (MADT) for effective offline learning. MADT leverages the transformer’s modelling ability for sequence modelling and integrates it seamlessly with both offline and online MARL tasks. A significant benefit of MADT is that it learns generalizable policies that can transfer between different types of agents under different task scenarios. On the StarCraft II offline dataset, MADT outperforms the state-of-the-art offline reinforcement learning (RL) baselines, including BCQ and CQL. When applied to online tasks, the pre-trained MADT significantly improves sample efficiency and enjoys strong performance in both few-short and zero-shot cases. To the best of our knowledge, this is the first work that studies and demonstrates the effectiveness of offline pre-trained models in terms of sample efficiency and generalizability enhancements for MARL.
The need for multimodal health data modeling: A practical approach for a federated-learning healthcare platform

Federated learning initiatives in healthcare are being developed to collaboratively train predictive models without the need to centralize sensitive personal data. GenoMed4All is one such project, with the goal of connecting European clinical and -omics data repositories on rare diseases through a federated learning platform. Currently, the consortium faces the challenge of a lack of well-established international datasets and interoperability standards for federated learning applications on rare diseases. This paper presents our practical approach to select and implement a Common Data Model (CDM) suitable for the federated training of predictive models applied to the medical domain, during the initial design phase of our federated learning platform. We describe our selection process, composed of identifying the consortium's needs, reviewing our functional and technical architecture specifications, and extracting a list of business requirements. We review the state of the art and evaluate three widely-used approaches (FHIR, OMOP and Phenopackets) based on a checklist of requirements and specifications. We discuss the pros and cons of each approach considering the use cases specific to our consortium as well as the generic issues of implementing a European federated learning healthcare platform. A list of lessons learned from the experience in our consortium is discussed, from the importance of establishing the proper communication channels for all stakeholders to technical aspects related to -omics data. For federated learning projects focused on secondary use of health data for predictive modeling, encompassing multiple data modalities, a phase of data model convergence is sorely needed to gather different data representations developed in the context of medical research, interoperability of clinical care software, imaging, and -omics analysis into a coherent, unified data model. Our work identifies this need and presents our experience and a list of actionable lessons learned for future work in this direction.
Deep Activity Recognition based on Patterns Discovery for Healthcare Monitoring

Healthcare monitoring for humans is important due to several factors including life quality and early detection of health-related problems. Human activity patterns recognition is the most promising ways to monitor human health. Uprisings in the human activity patterns recognition has enabled researchers to recognize multiple health issues through the usage of multiple sensory devices such as motion-based wearable sensors. Irrelevant motion patterns can lead to overlook the important activity recognition in daily living. For this purpose, an early discovery of motion patterns has been proposed for activity recognition in this paper. Main objective is to support the activity detection through motion patterns and deep learning mechanism. The proposed method contains three layered architecture including pre-processing layer, features engineering layer, and classification layer. The anticipated study is investigated over an openly available dataset named Opportunity and results have shown improvement in terms of achieving higher accuracy rate of 88.57%.
A Two-Stage Network Based on Transformer and Physical Model for Single Underwater Image Enhancement

The absorption and scattering properties of water can cause various distortions in underwater images, which limit the ability to investigate underwater resources. In this paper, we propose a two-stage network called WaterFormer to address this issue using deep learning and an underwater physical imaging model. The first stage of WaterFormer uses the Soft Reconstruction Network (SRN) to reconstruct underwater images based on the Jaffe–McGramery model, while the second stage uses the Hard Enhancement Network (HEN) to estimate the global residual between the original image and the reconstructed result to further enhance the images. To capture long dependencies between pixels, we designed the encoder and decoder of WaterFormer using the Transformer structure. Additionally, we propose the Locally Intended Multiple Layer Perceptron (LIMP) to help the network process local information more effectively, considering the significance of adjacent pixels in enhancing distorted underwater images. We also proposed the Channel-Wise Self-Attention module (CSA) to help the network learn more details of the distorted underwater images by considering the correlated and different distortions in RGB channels. To overcome the drawbacks of physical underwater image enhancement (UIE) methods, where extra errors are introduced when estimating multiple physical parameters separately, we proposed the Joint Parameter Estimation method (JPE). In this method, we integrated multiple parameters in the Jaffe–McGramery model into one joint parameter (JP) through a special mathematical transform, which allowed for physical reconstruction based on the joint parameter (JP). Our experimental results show that WaterFormer can effectively restore the color and texture details of underwater images in various underwater scenes with stable performance.