Reviewerly/exHarmony huggingface dataset:
Anomaly Detection for Hydroelectric Power Plants: a Machine Learning-based Approach

Hydroelectric is currently the most prominent among the sources of green energy, but, differently from the other sources, it has very strict requirements in terms of security that are taken into account with extremely robust constraints both at design and operations control times. In this paper, we evaluated the effectiveness of anomaly detection and explainability algorithms to supplement Decision Support System insights in Predictive Maintenance and Root Cause Analysis for hydroelectric power plants. The objective is to reduce operational costs and increase reliability in the plant, making hydroelectric technology more appealing to investors and promoting the transition to renewable energy. Specifically, the performance of several anomaly detection models was compared on real-world data with respect to the needs of the expert of the domain, that is the final user of the DSS, to work as an additional feature to speed up predictive maintenance. Additionally, the impact of SHapley Additive exPlanations values on helping the user understand the anomaly causes was investigated. Our findings are that the most performing algorithm was Auto-Encoder since it was able to find all recorded anomalies and even propose additional ones later confirmed by domain experts. The application of SHAP values was found to effectively guide the user toward the features related to the anomaly, although its application on streaming data was slow.
EXIF as Language: Learning Cross-Modal Associations between Images and Camera Metadata

We learn a visual representation that captures information about the camera that recorded a given photo. To do this, we train a multimodal embedding between image patches and the EXIF metadata that cameras automatically insert into image files. Our model represents this meta-data by simply converting it to text and then processing it with a transformer. The features that we learn significantly outperform other self-supervised and supervised features on downstream image forensics and calibration tasks. In particular, we successfully localize spliced image regions “zero shot” by clustering the visual embeddings for all of the patches within an image.
A Unified Knowledge Distillation Framework for Deep Directed Graphical Models

Knowledge distillation (KD) is a technique that transfers the knowledge from a large teacher network to a small student network. It has been widely applied to many different tasks, such as model compression and federated learning. However, existing KD methods fail to generalize to general deep directed graphical models (DGMs) with arbitrary layers of random variables. We refer by deep DGMs to DGMs whose conditional distributions are parameterized by deep neural networks. In this work, we propose a novel unified knowledge distillation framework for deep DGMs on various applications. Specifically, we leverage the reparameterization trick to hide the intermediate latent variables, resulting in a compact DGM. Then we develop a surrogate distillation loss to reduce error accumulation through multiple layers of random variables. Moreover, we present the connections between our method and some existing knowledge distillation approaches. The proposed framework is evaluated on four applications: data-free hierarchical variational autoencoder (VAE) compression, data-free variational recurrent neural networks (VRNN) compression, data-free Helmholtz Machine (HM) compression, and VAE continual learning. The results show that our distillation method out-performs the baselines in data-free model compression tasks. We further demonstrate that our method significantly improves the performance of KD-based continual learning for data generation. Our source code is available at https://github.com/YizhuoChen99/KD4DGM-CVPR.
You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model

Large-scale Transformer models bring significant improvements for various downstream vision language tasks with a unified architecture. The performance improvements come with increasing model size, resulting in slow inference speed and increased cost for severing. While some certain predictions benefit from the full computation of the large-scale model, not all of inputs need the same amount of computation to conduct, potentially leading to computation resource waste. To handle this challenge, early exiting is proposed to adaptively allocate computational power in term of input complexity to improve inference efficiency. The existing early exiting strategies usually adopt output confidence based on intermediate layers as a proxy of input complexity to incur the decision of skipping following layers. However, such strategies cannot be applied to encoder in the widely-used unified architecture with both encoder and decoder due to difficulty of output confidence estimation in the encoder layers. It is suboptimal in term of saving computation power to ignore the early exiting in encoder component. To address this issue, we propose a novel early exiting strategy for unified vision language models, which allows to dynamically skip the layers in encoder and decoder simultaneously in term of input layer-wise similarities with multiple times of early exiting, namely MuE. By decomposing the image and text modalities in the encoder, MuE is flexible and can skip different layers in term of modalities, advancing the inference efficiency while minimizing performance drop. Experiments on the SNLI-VE and MS COCO datasets show that the proposed approach MuE can reduce expected inference time by up to 50% and 40% while maintaining 99% and 96% performance respectively.
Behavioral Analysis of Vision-and-Language Navigation Agents

To be successful, Vision-and-Language Navigation (VLN) agents must be able to ground instructions to actions based on their surroundings. In this work, we develop a methodology to study agent behavior on a skill-specific basis - examining how well existing agents ground instructions about stopping, turning, and moving towards specified objects or rooms. Our approach is based on generating skill-specific interventions and measuring changes in agent predictions. We present a detailed case study analyzing the behavior of a recent agent and then compare multiple agents in terms of skill-specific competency scores. This analysis suggests that biases from training have lasting effects on agent behavior and that existing models are able to ground simple referring expressions. Our comparisons between models show that skill-specific scores correlate with improvements in overall VLN task performance.
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization

Prompt tuning is an effective way to adapt the pretrained visual-language model (VLM) to the downstream task using task-related textual tokens. Representative CoOp-based work combines the learnable textual tokens with the class tokens to obtain specific textual knowledge. However, the specific textual knowledge is worse generalization to the unseen classes because it forgets the essential general textual knowledge having a strong generalization ability. To tackle this issue, we introduce a novel Knowledge-guided Context Optimization (KgCoOp) to enhance the generalization ability of the learnable prompt for unseen classes. The key insight of KgCoOp is that the forgetting about essential knowledge can be alleviated by reducing the discrepancy between the learnable prompt and the hand-crafted prompt. Especially, KgCoOp minimizes the discrepancy between the textual embeddings generated by learned prompts and the hand-crafted prompts. Finally, adding the KgCoOp upon the contrastive loss can make a discriminative prompt for both seen and unseen tasks. Extensive evaluation of several benchmarks demonstrates that the proposed Knowledge-guided Context Optimization is an efficient method for prompt tuning, i.e., achieves better performance with less training time. code.
MAGVLT: Masked Generative Vision-and-Language Transformer

While generative modeling on multimodal image-text data has been actively developed with large-scale paired datasets, there have been limited attempts to generate both image and text data by a single model rather than a generation of one fixed modality conditioned on the other modality. In this paper, we explore a unified generative vision-and-language (VL) model that can produce both images and text sequences. Especially, we propose a generative VL transformer based on the non-autoregressive mask prediction, named MAGVLT, and compare it with an autoregressive generative VL transformer (ARGVLT). In comparison to ARGVLT, the proposed MAGVLT enables bidirectional context encoding, fast decoding by parallel token predictions in an iterative refinement, and extended editing capabilities such as image and text infilling. For rigorous training of our MAGVLT with image-text pairs from scratch, we combine the image-to-text, text-to-image, and joint image-and-text mask prediction tasks. Moreover, we devise two additional tasks based on the step-unrolled mask prediction and the selective prediction on the mixture of two image-text pairs. Experimental results on various downstream generation tasks of VL benchmarks show that our MAGVLT outperforms ARGVLT by a large margin even with significant inference speedup. Particularly, MAGVLT achieves competitive results on both zero-shot image-to-text and text-to-image generation tasks from MS-COCO by one moderate-sized model (fewer than 500M parameters) even without the use of monomodal data and networks.
Privacy in Local Energy Markets: A Framework for a Self-Sovereign Identity based P2P-Trading Authentication System

The energy sector is part of the critical infrastructure in modern society where security and privacy concerns of the customers of this infrastructure have been well studied and addressed. In the classical power grid, consumers are being supplied by a major power grid operator who has been government-sanctioned and therefore has to adhere to privacy and security guidelines. In recent years, however, with the shift to the usage of renewable energies, new concepts for the trading of energy flexibilities have emerged, that enable consumers with their own energy production to sell their spare energy back to the market. While reducing the load from the grid operators by enabling peer-to-peer trading between grid participants, this comes with new privacy concerns, which have to be addressed and regulated.This work proposes a conceptual framework based on Self-Sovereign Identity (SSI) to establish a secure and private peer-to-peer trading system between energy market participants, which provides the participants control over their own digital identity and what data these participants want to share with peers. Furthermore, this work discusses the proposal’s effects and implications on system security and user privacy.
Boundary-aware Backward-Compatible Representation via Adversarial Learning in Image Retrieval

Image retrieval plays an important role in the Internet world. Usually, the core parts of mainstream visual retrieval systems include an online service of the embedding model and a large-scale vector database. For traditional model upgrades, the old model will not be replaced by the new one until the embeddings of all the images in the database are re-computed by the new model, which takes days or weeks for a large amount of data. Recently, backward-compatible training (BCT) enables the new model to be immediately deployed online by making the new embeddings directly comparable to the old ones. For BCT, improving the compatibility of two models with less negative impact on retrieval performance is the key challenge. In this paper, we introduce AdvBCT, an Adversarial Backward-Compatible Training method with an elastic boundary constraint that takes both compatibility and discrimination into consideration. We first employ adversarial learning to minimize the distribution disparity between embeddings of the new model and the old model. Meanwhile, we add an elastic boundary constraint during training to improve compatibility and discrimination efficiently. Extensive experiments on GLDv2, Revisited Oxford (ROxford), and Revisited Paris (RParis) demonstrate that our method outperforms other BCT methods on both compatibility and discrimination. The implementation of AdvBCT will be publicly available at https://github.com/Ashespt/AdvBCT.
Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning

Compositional Zero-Shot Learning (CZSL) aims to recognize novel concepts formed by known states and objects during training.Existing methods either learn the combined state-object representation, challenging the generalization of unseen compositions, or design two classifiers to identify state and object separately from image features, ignoring the intrinsic relationship between them.To jointly eliminate the above issues and construct a more robust CZSL system, we propose a novel framework termed Decomposed Fusion with Soft Prompt (DFSP) 1 , by involving vision-language models (VLMs) for unseen composition recognition.Specifically, DFSP constructs a vector combination of learnable soft prompts with state and object to establish the joint representation of them.In addition, a cross-modal decomposed fusion module is designed between the language and image branches, which decomposes state and object among language features instead of image features.Notably, being fused with the decomposed features, the image features can be more expressive for learning the relationship with states and objects, respectively, to improve the response of unseen compositions in the pair space, hence narrowing the domain gap between seen and unseen sets.Experimental results on three challenging benchmarks demonstrate that our approach significantly outperforms other state-of-theart methods by large margins.
Simulated Annealing in Early Layers Leads to Better Generalization

Recently, a number of iterative learning methods have been introduced to improve generalization. These typically rely on training for longer periods of time in exchange for improved generalization. LLF (later-layer-forgetting) is a state-of-the-art method in this category. It strengthens learning in early layers by periodically re-initializing the last few layers of the network. Our principal innovation in this work is to use Simulated annealing in EArly Layers (SEAL) of the network in place of re-initialization of later layers. Essentially, later layers go through the normal gradient descent process, while the early layers go through short stints of gradient ascent followed by gradient descent. Extensive experiments on the popular Tiny-ImageNet dataset benchmark and a series of transfer learning and few-shot learning tasks show that we outperform LLF by a significant margin. We further show that, compared to normal training, LLF features, although improving on the target task, degrade the transfer learning performance across all datasets we explored. In comparison, our method outperforms LLF across the same target datasets by a large margin. We also show that the prediction depth of our method is significantly lower than that of LLF and normal training, indicating on average better prediction performance. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> The code to reproduce our results is publicly available at: https://github.com/amiiir-sarfi/SEAL
Convolutional Neural Networks on the Edge: A Comparison Between FPGA and GPU

With more computation tasks being pushed towards the edge, it becomes increasingly challenging to decide the right hardware approach for running edge AI tasks. Besides customized AI accelerators, existing hardware platforms such as FPGAs have been employed to support edge computing. Recently, small GPUs have also been developed to run AI tasks that are tailored for edge. While each platform exhibits its own advantages, the shortcomings are also identified. In this work, we present a study where a head to head comparison is conducted by running the same CNN on two popular edge platforms, FPGAs and GPUs. We compare multiple dimensions such as power, inference speed, ease of development and accuracy. The comparison results can provide an initial guidance for edge computing developers who don’t have immediate access to both platforms.
Hierarchical Supervision and Shuffle Data Augmentation for 3D Semi-Supervised Object Detection

State-of-the-art 3D object detectors are usually trained on large-scale datasets with high-quality 3D annotations. However, such 3D annotations are often expensive and time-consuming, which may not be practical for real applications. A natural remedy is to adopt semi-supervised learning (SSL) by leveraging a limited amount of labeled samples and abundant unlabeled samples. Current pseudo-labeling-based SSL object detection methods mainly adopt a teacher-student framework, with a single fixed threshold strategy to generate supervision signals, which inevitably brings confused supervision when guiding the student network training. Besides, the data augmentation of the point cloud in the typical teacher-student framework is too weak, and only contains basic down sampling and flip-and-shift (i.e., rotate and scaling), which hinders the effective learning of feature information. Hence, we address these issues by introducing a novel approach of Hierarchical Supervision and Shuffle Data Augmentation (HSSDA), which is a simple yet effective teacher-student framework. The teacher network generates more reasonable supervision for the student network by designing a dynamic dual-threshold strategy. Besides, the shuffle data augmentation strategy is designed to strengthen the feature representation ability of the student network. Extensive experiments show that HSSDA consistently outperforms the recent state-of-the-art methods on different datasets. The code will be released at https://github.com/azhuantou/HSSDA.
Implementation of the Harmonic Extreme Learning Machine for Clustering the Solar Irradiance in Bangkok, Thailand

Solar irradiance is renewable energy that can quickly produce energy for households and industries. On each day and event, the solar irradiance can be varied due to the change in weather. The solar irradiance clustering can evaluate the appropriate generation pattern of a solar photovoltaic system for analyzing daily energy generation. Therefore, this paper proposes a novel clustering method to evaluate the appropriate cluster of solar irradiance generation by utilizing the Harmonic Extreme Learning Machine model to improve the performance of solar irradiance analysis and management. The experimental results showed that the Harmonic Extreme Learning Machine model had given the minimum total sum of distances between the centroid cluster and the represented data. The plot of the result is evident for explaining and interpreting the behavior of solar irradiance data and managing the data to the appropriate cluster.
What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging

Inferring past human motion from RGB images is challenging due to the inherent uncertainty of the prediction problem. Thermal images, on the other hand, encode traces of past human-object interactions left in the environment via thermal radiation measurement. Based on this observation, we collect the first RGB-Thermal dataset for human motion analysis, dubbed Thermal-1M. Then we develop a three-stage neural network model for accurate past human pose estimation. Comprehensive experiments show that thermal cues significantly reduce the ambiguities of this task, and the proposed model achieves remarkable performance. The dataset is available at https://github.com/ZitianTang/Thermal-IM.
DeLMS: A decentralized learning management system using Ethereum smart contracts and IPFS

A learning management system (LMS) is a web-based technology developed to improve the learning process in educational institutions. This study addresses the drawbacks of existing LMS by developing DeLMS (Decentralized Learning Management System) and aims to revolutionize the way educational institutions manage and deliver educational content using Ethereum smart contracts and the IPFS (InterPlanetary File System). By leveraging Smart contracts and IPFS cutting-edge technologies, DeLMS provides a secure, transparent and decentralized platform for managing and delivering educational content. The content addressing feature of IPFS eliminates the need for constant requests to centralized servers for the location of files, resulting in faster file transfer and reduced energy consumption. The proposed system has the potential to change the way educational institutions operate, making it more accessible and beneficial for both students and educators.
Parameter Efficient Local Implicit Image Function Network for Face Segmentation

Face parsing is defined as the per-pixel labeling of images containing human faces. The labels are defined to identify key facial regions like eyes, lips, nose, hair, etc. In this work, we make use of the structural consistency of the human face to propose a lightweight face-parsing method using a Local Implicit Function network, FP-LIIF. We propose a simple architecture having a convolutional encoder and a pixel MLP decoder that uses 1/26 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> number of parameters compared to the state-of-the-art models and yet matches or outperforms state-of-the-art models on multiple datasets, like CelebAMask-HQ and LaPa. We do not use any pretraining, and compared to other works, our network can also generate segmentation at different resolutions without any changes in the input resolution. This work enables the use of facial segmentation on low-compute or low-bandwidth devices because of its higher FPS and smaller model size.
A Design Approach and Prototype Implementation for Factory Monitoring Based on Virtual and Augmented Reality at the Edge of Industry 4.0

Virtual and augmented reality are currently enjoying a great deal of attention from the research community and the industry towards their adoption within industrial spaces and processes. However, the current design and implementation landscape is still very fluid, while the community as a whole has not yet consolidated into concrete design directions, other than basic patterns. Other open issues include the choice over a cloud or edge-based architecture when designing such systems. Within this work, we present our approach for a monitoring intervention inside a factory space utilizing both Virtual Reality (VR) and Augmented Reality (AR), based primarily on edge computing, while also utilizing the cloud. We discuss its main design directions, as well as a basic ontology to aid in simple description of factory assets. In order to highlight the design aspects of our approach, we present a prototype implementation, based on a use case scenario in a factory site, within the context of the EnerMan H2020 project.