Reviewerly/exHarmony huggingface dataset:
Lifelong Change Detection: Continuous Domain Adaptation for Small Object Change Detection in Everyday Robot Navigation

The recently emerging research area in robotics, ground view change detection, suffers from its ill-posed-ness because of visual uncertainty combined with complex nonlinear perspective projection. To regularize the ill-posed-ness, the commonly applied supervised learning methods (e.g., CSCD-Net) rely on manually annotated high-quality object-class-specific priors. In this work, we consider general application domains where no manual annotation is available and present a fully self-supervised approach. The proposed approach adopts the powerful and versatile idea that object changes detected during everyday robot navigation can be reused as additional priors to improve future change detection tasks. Furthermore, a robustified framework is implemented and verified experimentally in a new challenging practical application scenario: ground-view small object change detection.
Combining Knowledge Distillation and Transfer Learning for Sensor Fusion in Visible and Thermal Camera-based Person Classification

Visible and thermal camera-based sensor fusion has shown to address the limitations and enhance the robustness of visible camera-based person classification. In this paper, we propose to further enhance the classification accuracy of visible-thermal person classification using transfer learning, knowledge distillation, and the vision transformer. In our work, the visible-thermal person classifier is implemented using the vision transformer. The proposed classifier is trained using the transfer learning and knowledge distillation techniques. To train the proposed classifier, visible and thermal teacher models are implemented using the vision transformers. The multimodal classifier learns from the two teachers using a novel loss function which incorporates the knowledge distillation. The proposed method is validated on the public Speaking Faces dataset. A comparative analysis with baseline algorithms and an ablation study is performed. The results show that the proposed framework reports an enhanced classification accuracy.
Intra-frame Skeleton Constraints Modeling and Grouping Strategy Based Multi-Scale Graph Convolution Network for 3D Human Motion Prediction

Attention-based feed-forward networks and graph convolution networks have recently shown great promise in 3D skeleton-based human motion prediction for their good performance in learning temporal and spatial relations. However, previous methods have two critical issues: first, spatial dependencies for distal joints in each independent frame are hard to learn; second, the basic architecture of graph convolution network ignores hierarchical structure and diverse motion patterns of different body parts. To address these issues, this paper proposes an intra-frame skeleton constraints modeling method and a Grouping based Multi-Scale Graph Convolution Network (GMS-GCN) model. The intra-frame skeleton constraints modeling method leverages self-attention mechanism and a designed adjacency matrix to model the skeleton constraints of distal joints in each independent frame. The GMS-GCN utilizes a grouping strategy to learn the dynamics of various body parts separately. Instead of mapping features in the same feature space, GMS-GCN extracts human body features in different dimensions by up-sample and down-sample GCN layers. Experiment results demonstrate that our method achieves an average MPJPE of 34.7mm for short-term prediction and 93.2mm for long-term prediction and both outperform the state-of-the-art approaches.
Outline Generation Transformer for Bilingual Scene Text Recognition

We propose the Outline Generation Transformer (OGT) for bilingual Scene Text Recognition (STR). As most STR approaches focus on English, we consider both English and Chinese as Chinese is also a major language, and it is a common scene in many areas/countries where both languages can be seen. The OGT consists of an Outline Generator (OG) and a transformer with a language model embedded. The OG detects the character outline of the text and embeds the outline features into a transformer with the outline-query cross-attention layer to better locate each character and enhance the text recognition performance. The training of OGT has two phases, one is training on synthetic data where the text outline masks are made available, followed by the other training on real data where the text outline masks can only be estimated. The proposed OGT is evaluated on several benchmark datasets and compared with state-of-the-art methods.
Generalizable Solar Irradiation Prediction using Large Transformer Models with Sky Imagery

Deployment of solar power system in new locations impose several challenges on the operations of local and regional power grids due to the inherent variation in ground-level solar irradiance. This work proposes a novel real-time solar now-casting methodology for solar irradiance prediction based on deep transfer learning from ground-based sky imagery. Existing approaches use statistical methods or Convolutional Neural Networks for irradiation regression trained for a particular location that cannot be transferred to new locations deploying potentially different imaging sensors. This observation motivated us to introduce a large deep neural network based on Vision Transformers that is generalizable and transferable to different scenarios.The system is developed using multiple years of solar irradiance and sky image recordings in two locations. We captured our own data set in Princeton, NJ, USA and also used open-source ASI16 benchmark dataset captured in Golden, CO, USA. The method is validated against these two locations of diverse geographic, climatic conditions and sensor variation. Results show that the proposed method is robust and highly accurate (85-90% accuracy) for multiple locations deployment with 50% less data requirement from new locations.
Black-box Adversarial Attack against Visual Interpreters for Deep Neural Networks

With the rapid development of deep neural networks (DNNs), eXplainable AI, which provides a basis for prediction on inputs, has become increasingly important. In addition, DNNs have a vulnerability called an Adversarial Example (AE), which can cause incorrect output by applying special perturbations to inputs. Potential vulnerabilities can also exist in image interpreters such as GradCAM, necessitating their investigation, as these vulnerabilities could potentially result in misdiagnosis within medical imaging. Therefore, this study proposes a black-box adversarial attack method that misleads the image interpreter using Sep-CMA-ES. The proposed method deceptively shifts the focus area of the image interpreter to a different location from that of the original image while maintaining the same predictive labels.
Padding Investigations for CNNs in Scene Parsing Tasks

Zero padding is widely used in convolutional neural networks (CNNs) to prevent the size of feature maps diminishing too fast. However, it has been claimed to disturb the statistics at the border [9]. In this work, we compare various padding methods for the scene parsing task and propose an alternative padding method (CApadding) by extending the image to alleviate the border issue. Experiments on Cityspaces [2] and Deep-Globe [3] show that models with the proposed padding method achieves higher mean Intersection-Over-Union (IoU) than the zero padding based models.
Uncertainty Criteria in Active Transfer Learning for Efficient Video-Specific Human Pose Estimation

This paper presents a combination of Active Learning (AL) and Transfer Learning (TL) for efficiently adapting Human Pose (HP) estimators to individual videos. The proposed approach quantifies estimation uncertainty through the temporal changes and unnaturalness of estimated HPs. These uncertainty criteria are combined with clustering-based representativeness criterion to avoid the useless selection of similar samples. Experiments demonstrated that the proposed method achieves high learning efficiency and outperforms comparative methods.
Using Unconditional Diffusion Models in Level Generation for Super Mario Bros

This study introduces a novel methodology for generating levels in the iconic video game Super Mario Bros. using a diffusion model based on a UNet architecture. The model is trained on existing levels, represented as a categorical distribution, to accurately capture the game’s fundamental mechanics and design principles. The proposed approach demonstrates notable success in producing high-quality and diverse levels, with a significant proportion being playable by an artificial agent. This research emphasizes the potential of diffusion models as an eﬃcient tool for procedural content generation and highlights their potential impact on the development of new video games and the enhancement of existing games through generated content.
PALF: Pre-Annotation and Camera-LiDAR Late Fusion for the Easy Annotation of Point Clouds

3D object detection has become indispensable in the field of autonomous driving. To date, gratifying breakthroughs have been recorded in 3D object detection research, attributed to deep learning. However, deep learning algorithms are data-driven and require large amounts of annotated point cloud data for training and evaluation. Unlike 2D images, annotating point cloud data is difficult due to the limitations of sparsity, irregularity, and low resolution, which requires more manual work, and the annotation efficiency is much lower than annotating 2D images. Therefore, we propose an annotation algorithm for point cloud data, which is pre-annotation and camera-LiDAR late fusion algorithm to annotate easily and accurately.
Combining Static Specular Flow and Highlight with Deep Features for Specular Surface Detection

To apply robot teaching to a factory with many mirror-polished parts, it is necessary to detect the mirror-like surface accurately. Deep models for mirror detection have been studied by designing mirror-specific features, e.g., contextual contrast and similarity. However, the mirror-polished parts, e.g., plastic molds, tend to have complex shapes and ambiguous boundaries, and thus existing mirror-specific deep features could not work well. To detect such complex mirror-like surfaces, we propose combining static specular flow and highlight, frequently appearing in specular surfaces, with deep model-based multi-level feature pyramids and adaptively integrating multiple feature maps, including mirror-specific ones. Through experiments with our original real-world plastic mold dataset, we show the effectiveness of the proposed method.
Word to Sentence Visual Semantic Similarity for Caption Generation: Lessons Learned

This paper focuses on enhancing the captions generated by image captioning systems. We propose an approach for improving caption generation systems by choosing the most closely related output to the image rather than the most likely output produced by the model. Our model revises the language generation output beam search from a visual context perspective. We employ a visual semantic measure in a word and sentence level manner to match the proper caption to the related information in the image. This approach can be applied to any caption system as a post-processing method.
Domain Adaptation from Visible-Light to FIR with Reliable Pseudo Labels

Deep learning object detection models using visible-light cameras are easily affected by weather and lighting conditions, whereas those using far-infrared cameras are less affected by such conditions. This paper proposes a domain adaptation method using pseudo labels from a visible-light camera toward an accurate object detection from far-infrared images. Our method projects visible light-domain detection results onto far-infrared images, and uses them as pseudo labels for training a far-infrared detection model. We confirmed the effectiveness of our method through experiments.
Shape Preservation in Image Style Transfer for Gaze Estimation

This paper proposes image style transfer with shape preservation for gaze estimation. While several shape preservation constraints are proposed, we present additional shape preservation constraints using (i) dense pixelwise correspondences between the original and its transferred images and (ii) task-driven learning using gaze estimation error for directly improving gaze direction estimation. A variety of experiments with other SOTA methods, publicly-available datasets, and ablation studies validate the effectiveness of our method.
YOLOv5 with Mixed Backbone for Efficient Spatio-Temporal Hand Gesture Localization and Recognition

Spatio-temporal Hand Gesture Localization and Recognition (SHGLR) refers to analyzing the spatial and temporal aspects of hand movements for detecting and identifying hand gestures in a video. Current state-of-the-art approaches for SHGLR utilize large and complex architectures that result in a high computational cost. To address this issue, we present a new efficient method based on a mixed backbone for YOLOv5. We decided to use it since it is a lightweight and one-stage framework. We designed a mixed backbone that combines 2D and 3D convolutions to obtain temporal information from previous frames. The proposed method offers an efficient way to perform SHGLR on videos by inflating specific convolutions of the backbone while keeping a similar computational cost to the conventional YOLOv5. Due to its challenging and continuous hand gestures, we conduct experiments using the IPN Hand dataset. Our proposed method achieves a frame mAP@0.5 of 66.52% with a 6-frame clip input, outperforming conventional YOLOv5 by 7.89%, demonstrating the effectiveness of our approach.
A Scalable Clustered Architecture for Cyber-Physical Systems

Developing distributed and scalable Cyber-Physical Systems (CPS) that can handle large amounts of data at high data rates at the edge, remains a challenging task. Also, the limited availability of open-source solutions makes it difficult for developers and researchers to experiment with and deploy CPSs on a larger scale. This work introduces Edge4CPS, an open-source multi-architecture solution built over Kubernetes that aims to enable an easy to use, efficient and scalable solution for the deployment of applications on edge-like distributed computing clusters. To verify the successful real-world implementation of the introduced architecture, the system was tested in a railway scenario, derived from the Ferrovia 4.0 project, which highlights its functionalities.
SiD<sup>2</sup>Re - A novel simulation framework for drifting regression data

Applying supervised machine learning techniques to data streams in non-stationary environments is a challenge that has to be overcome in order to exploit the potentials of machine learning in the context of industrial production. Particularly for regression learning tasks, there is a major need for novel methods to overcome these challenges. In this work we formalize the understanding of data drifts and concept drifts and explain how changes in the process environment can lead to a degradation in model performance. Further, we introduce the Simulator of Drifting Data in Regression problem (SiD <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> Re) for generating benchmark datasets laying the foundation for comparing new algorithms for drift detection and drift adaptation with respect to various drift characteristics.
Graph Neural Network-Based Measurement Inference on Irregular Sensor Geometries

In general, Graph Neural Networks (GNNs) enable modeling and learning in the context of complex data like graphs and time series – which is typically difficult for standard Deep Learning approaches. This paper proposes an approach for modeling a prediction task using irregularly sampled spatio-temporal sensor data via GNNs. Specifically, we present a method for modeling spatio-temporal sensor data represented as graphs, and a more convenient image representation enabling standard convolutional deep learning. By mapping the irregularly sampled graph to a regular graph representation, we can then integrate temporal sensor information with spatial information. In our experimentation, we demonstrate the efficacy of the proposed approach in inspection contexts of oil and gas pipelines.
Building a small and informative phylogenetic supertree

We combine two fundamental optimization problems related to the construction of phylogenetic trees called maximum rooted triplets consistency and minimally resolved supertree into a new problem, which we call q-maximum rooted triplets consistency (q-MAXRTC). It takes as input a set R of rooted, binary phylogenetic trees with three leaves each and asks for a phylogenetic tree with exactly q internal nodes that contains the largest possible number of trees from R. We prove that q-MAXRTC is NP-hard to approximate within a constant, develop polynomial-time approximation algorithms for different values of q, and show experimentally that representing a phylogenetic tree by one having much fewer nodes typically does not destroy too much branching information. To demonstrate the algorithmic advantage of using trees with few internal nodes, we also propose a new algorithm for computing the rooted triplet distance that is faster than the existing algorithms when restricted to such trees.
Fractional Polynomial Models as Special Cases of Bayesian Generalized Nonlinear Models

We propose a framework for fitting multivariable fractional polynomial models as special cases of Bayesian generalized nonlinear models, applying an adapted version of the genetically modified mode jumping Markov chain Monte Carlo algorithm. The universality of the Bayesian generalized nonlinear models allows us to employ a Bayesian version of fractional polynomials in any supervised learning task, including regression, classification, and time-to-event data analysis. We show through a simulation study that our novel approach performs similarly to the classical frequentist multivariable fractional polynomials approach in terms of variable selection, identification of the true functional forms, and prediction ability, while naturally providing, in contrast to its frequentist version, a coherent inference framework. Real-data examples provide further evidence in favor of our approach and show its flexibility.
A Benchmark for Maritime Object Detection with Centernet on an Improved Dataset, ABOships-PLUS

Object detection from waterborne imagery is an essential aspect in maritime traffic management, navigation safety and coastal security. Building efficient autonomous systems, which can take decisions in critical situations, requires an integration of complex object detectors in real time. Object detectors trained on generic datasets often give unsatisfactory results in complex scenarios like the maritime environment, since only a fraction of their images contain maritime vessels. Publicly available domain-specific datasets are scarce, and they are limited in the number of images and annotations. Compared to object detection in applications such as autonomous vehicles, maritime vessel detection is considerably reduced in computer vision research. This creates a deficit in exhaustive benchmarking studies for maritime detection datasets. To bridge this gap, we relabel the ABOships dataset and benchmark a state-of-the-art center-based detector, Centernet, on the newly relabeled dataset, ABOships-PLUS. We explore its performance under different feature extractors, and investigate the effect of object size and inter-class variation on detection accuracy. The reported benchmarking illustrates that the ABOships-PLUS dataset is adequate to use in supervised domain adaptation. The experimental results show that Centernet with DLA (Deep Layer Aggregation) as a feature extractor achieved the highest accuracy in detecting maritime objects overall (with mean average precision 74.4%).
Influence Factors on User Manual Engagement in the Context of Smart Wearable Devices

The emergence of smart wearable devices has gradually introduced a vast array of functions that may greatly serve the everyday needs of users. Although their ever-increasing list of capabilities may be compensated for by intuitive design, users may still have numerous reasons to engage with their user manuals. In this paper, we present four research efforts on the user manuals of smart wearable devices. Our work tackles various influence factors, investigates user behavior, and studies how the device itself affects user manual engagement, as well as the other way around. The research efforts address user experience and behavior, personal preference, device attributes, and the performance metrics of user manuals. We specifically studied smartwatches and smart bands, and explored engagement with medical-purpose smart wearables and their user manuals. As a novelty of our contribution, we classify test participants as regular users, professional users, and technical writers, based on their connection to user manuals, and separately analyze their responses to our questionnaire-based research efforts. The obtained results indicate that user experience and device complexity as influence factors have a statistically significant impact on user manual engagement for regular users, and highlight how regular users and technical writers differ in preference and search-related tasks.
Crowd-Sourced Buildings Data Collection and Remote Training: New Opportunities to Engage Students in Seismic Risk Reduction

Young generations are increasingly committed to understanding disasters, and are a key player in current and future disaster risk reduction activities. The availability of online tools opened new perspectives in the organization of risk-related educational activities, in particular in earthquake-prone areas. This is the case of CEDAS (building CEnsus for seismic Damage Assessment), a pilot training activity aimed at collecting risk-related information while educating high-school students about seismic risk. During this experimental activity, students collected and elaborated crowdsourced data on the main building typologies in the proximity of their homes. In a few months, students created a dataset of valuable risk-related information, while getting familiar with the area where they live. Data collection was performed both on-site, using smartphones, and online, based on remote sensing images provided by multiple sources (e.g., Google maps and street view). This allowed all students, including those with limited mobility, to perform the activity. The CEDAS experience pointed out the potential of online tools and remote sensing images, combined with practical activities and basic training in exploratory data analysis, to engage students in an inclusive way. The proposed approach can be naturally expanded in a multi-risk perspective, and can be adjusted, eventually increasing the technical content of collected information, to the specific training and expertise of the involved students, from high-school to university level.