Reviewerly/exHarmony huggingface dataset:
MIPI 2023 Challenge on Nighttime Flare Removal: Methods and Results

Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). With the success of the 1st MIPI Workshop@ECCV 2022, we introduce the second MIPI challenge including four tracks focusing on novel image sensors and imaging algorithms. In this paper, we summarize and review the Nighttime Flare Removal track on MIPI 2023. In total, 120 participants were successfully registered, and 11 teams submitted results in the final testing phase. The developed solutions in this challenge achieved state-of-the-art performance on Nighttime Flare Removal. A detailed description of all models developed in this challenge is provided in this paper. More details of this challenge and the link to the dataset can be found at https://mipi-challenge.org/MIPI2023/.
The MONET dataset: Multimodal drone thermal dataset recorded in rural scenarios

We present MONET, a new multimodal dataset captured using a thermal camera mounted on a drone that flew over rural areas, and recorded human and vehicle activities. We captured MONET to study the problem of object localisation and behaviour understanding of targets undergoing large-scale variations and being recorded from different and moving viewpoints. Target activities occur in two different land sites, each with unique scene structures and cluttered backgrounds. MONET consists of approximately 53K images featuring 162K manually annotated bounding boxes. Each image is timestamp-aligned with drone metadata that includes information about attitudes, speed, altitude, and GPS coordinates. MONET is different from previous thermal drone datasets because it features multimodal data, including rural scenes captured with thermal cameras containing both person and vehicle targets, along with trajectory information and metadata. We assessed the difficulty of the dataset in terms of transfer learning between the two sites and evaluated nine object detection algorithms to identify the open challenges associated with this type of data. Project page: https://github.com/fabiopoiesi/monet_dataset.
Pyramid Ensemble Structure for High Resolution Image Shadow Removal

Existing methods for shadow removal in high-resolution images may not be effective due to challenges such as the time-consuming nature of training and the loss of visual data during image cropping or resizing, highlighting the necessity for the development of more efficient methods. In this paper, we propose a novel Pyramid Ensemble Structure (PES) for High Resolution Image Shadow Removal. Our approach takes advantage of multiple scales by constructing pyramid inputs that allow for the capturing of a wide range of shadow sizes and shapes. We then train the network in pyramid stages to enhance global information processing. Furthermore, an ensemble of different shadow removal models is employed, and the maximum value is chosen to indicate the least amount of remaining shadow in the output. Experiments on both validation and testing data sets confirm the effectiveness of our method. In the Image Shadow Removal Challenge competition, our method obtained 22.36 PSNR score (1st place) and 0.70 SSIM score (2nd place) on the test sets.
Quality assessment of enhanced videos guided by aesthetics and technical quality attributes

In this work we propose a novel method to evaluate the quality of enhanced videos. Perceived quality of a video depends on both technical aspects, such as the presence of distortions like noise and blur, and non-technical factors, such as content preference and recommendation. Our approach involves the use of three deep learning based models that encode video sequences in terms of their overall technical quality, quality-related attributes, and aesthetic quality. The resulting feature vectors are adaptively combined and used as input to a Support Vector Regressor to estimate the video quality score. Quantitative results on the recently released VQA Dataset for Perceptual Video Enhancement (VDPVE) introduced for the NTIRE 2023 Quality Assessment of Video Enhancement Challenge demonstrates the effectiveness of the proposed method.
Fashion-Specific Ambiguous Expression Interpretation with Partial Visual-Semantic Embedding

A novel technology named fashion intelligence system has been proposed to quantify ambiguous expressions unique to fashion, such as "casual," "adult-casual," and "office-casual," and to support users’ understanding of fashion. However, the existing visual-semantic embedding (VSE) model, which is the basis of its system, does not support situations in which images are composed of multiple parts such as hair, tops, pants, skirts, and shoes. We propose partial VSE, which enables sensitive learning for each part of the fashion outfits. This enables five types of practical functionalities, particularly image-retrieval tasks in which changes are made only to the specified parts and image-reordering tasks that focus on the specified parts by the single model. Based on both the multiple unique qualitative and quantitative evaluation experiments, we show the effectiveness of the proposed model.
Face Recognition Accuracy Across Demographics: Shining a Light Into the Problem

We explore varying face recognition accuracy across demographic groups as a phenomenon partly caused by differences in face illumination. We observe that for a common operational scenario with controlled image acquisition, there is a large difference in face region brightness between African-American and Caucasian, and also a smaller difference between male and female. We show that impostor image pairs with both faces under-exposed, or both overexposed, have an increased false match rate (FMR). Conversely, image pairs with strongly different face brightness have a decreased similarity measure. We propose a brightness information metric to measure variation in brightness in the face and show that face brightness that is too low or too high has reduced information in the face region, providing a cause for the lower accuracy. Based on this, for operational scenarios with controlled image acquisition, illumination should be adjusted for each individual to obtain appropriate face image brightness. This is the first work that we are aware of to explore how the level of brightness of the skin region in a pair of face images (rather than a single image) impacts face recognition accuracy, and to evaluate this as a systematic factor causing unequal accuracy across demographics. The code is at https://github.com/HaiyuWu/FaceBrightness
Interpretable Model-Agnostic Plausibility Verification for 2D Object Detectors Using Domain-Invariant Concept Bottleneck Models

Despite the unchallenged performance, deep neural network (DNN) based object detectors (OD) for computer vision have inherent, hard-to-verify limitations like brittleness, opacity, and unknown behavior on corner cases. Therefore, operation-time safety measures like monitors will be inevitable—even mandatory—for use in safety-critical applications like automated driving (AD). This paper presents an approach for plausibilization of OD detections using a small model-agnostic, robust, interpretable, and domain-invariant image classification model. The safety requirements of interpretability and robustness are achieved by using a small concept bottleneck model (CBM), a DNN intercepted by interpretable intermediate outputs. The domain-invariance is necessary for robustness against common domain shifts, and for cheap adaptation to diverse AD settings. While vanilla CBMs are here shown to fail in case of domain shifts like natural perturbations, we substantially improve the CBM via combination with trainable color-invariance filters developed for domain adaptation. Furthermore, the monitor that utilizes CBMs with trainable color-invarince filters is successfully applied in an AD OD setting for detection of hallucinated objects with zero-shot domain adaptation, and to false positive detection with fewshot adaptation, proving this to be a promising approach for error monitoring.
Monocular 3D Human Pose Estimation for Sports Broadcasts using Partial Sports Field Registration

The filming of sporting events projects and flattens the movement of athletes in the world onto a 2D broadcast image. The pixel locations of joints in these images can be detected with high validity. Recovering the actual 3D movement of the limbs (kinematics) of the athletes requires lifting these 2D pixel locations back into a third dimension, implying a certain scene geometry. The well-known line markings of sports fields allow for the calibration of the camera and for determining the actual geometry of the scene. Close-up shots of athletes are required to extract detailed kinematics, which in turn obfuscates the pertinent field markers for camera calibration. We suggest partial sports field registration, which determines a set of scene-consistent camera calibrations up to a single degree of freedom. Through joint optimization of 3D pose estimation and camera calibration, we demonstrate the successful extraction of 3D running kinematics on a 400m track. In this work, we combine advances in 2D human pose estimation and camera calibration via partial sports field registration to demonstrate an avenue for collecting valid large-scale kinematic datasets. We generate a synthetic dataset of more than 10k images in Unreal Engine 5 with different viewpoints, running styles, and body types, to show the limitations of existing monocular 3D HPE methods. Synthetic data and code are available at https://github.com/tobibaum/PartialSportsFieldReg_3DHPE.
A Unified Transformer-based Tracker for Anti-UAV Tracking

Recently, the need for advanced anti-UAV techniques is increasing due to the rising threat of unauthorized drone intrusion. Object tracking, specifically in thermal infrared (TIR) videos, offers a potential solution to this issue. However, the tracked target often suffers dramatic scale variation, frequent target disappearance, and camera movement which severely influence tracking performance. Therefore, we propose a Unified Transformer-based Tracker, dubbed UTTracker, which contains the following four modules. Firstly, a multi-region local tracking module is designed with temporal cues for tackling target appearance variation and multi-region search for tracking targets in multi proposals. Complementarily, a global detection module is introduced to meet the challenge of target frequent disappearance. Meanwhile, a background correction module is incorporated to align the backgrounds between adjacent frames for alleviating camera movement. Particularly, a dynamic small object detection module for tracking the small target that lacks appearance information. Thanks to the designed modules, our UTTracker can achieve robust UAV tracking in TIR scenarios. Numerous experiments on the 1st and 2nd anti-UAV benchmarks demonstrate the effectiveness of UTTracker. Notably, UTTracker is the foundation of the 2nd-place winning entry in the 3rd Anti-UAV Challenge.
Few-Shot Depth Completion Using Denoising Diffusion Probabilistic Model

Generating dense depth maps from sparse LiDAR data is a challenging task, benefiting a lot of computer vision and photogrammetry tasks including autonomous driving, 3D point cloud generation, and aerial spatial awareness. Using RGB images as guidance to generate pixel-wise depth map is good, but these multi-modal data fusion networks always need numerous high-quality datasets like KITTI dataset to train on. Since this may be difficult in some cases, how to achieve few-shot learning with less train samples is worth discussing. So in this paper, we firstly proposed a few-shot learning paradigm for depth completion based on pre-trained denoising diffusion probabilistic model. To evaluate our model and other baselines, we constructed a smaller train set with only 12.5% samples from KITTI depth completion dataset to test their few-shot learning ability. Our model achieved the best on all metrics with a 5% improvement in RMSE compared to the second-place model.
Hardware-Aware Pruning for FPGA Deep Learning Accelerators

Pruning has been widely used for deep neural network optimization and compression. In this paper we propose a pruning method to accelerate FPGA implementations of neural networks with the goal of lowering the inference time as much as possible, while taking into account hardware-specific constraints. We use the normalized L2-norm as a measure of filter importance and iteratively prune the network with a predefined pruning stepsize. We extend this method to also prune around residual connections using the maximum normalized L2-norm as a representation of importance for a group of connected channels. We introduce a hardware-aware pruning method for FPGA deep learning accelerators, adaptively pruning the neural network based on the size of the systolic array used to calculate the convolutions. We validate our methods by pruning a model, designed for polyp segmentation in colonoscopy videos, on two different datasets. This results in almost halving the inference time with minimal loss of accuracy on both datasets. We prove that our two contributions yield an extra 30% increase in processing speed compared to classical L2 pruning.
AFLChain: Blockchain-enabled Asynchronous Federated Learning in Edge Computing Network

Edge computing network (ECN), which could process learning tasks at the edge, is considered as a potential solution to release the burden of the cloud. Meanwhile, to protect user privacy, federated learning (FL) is used in the ECN to establish models by multi-party collaborative learning on numbers of edge nodes (ENs). However, due to the frequent data interaction between the cloud server and distributed ENs, the reliability of data transmission and the privacy protection capability of the network cannot be guaranteed. In this paper, a distributed ECN is considered, to improve the learning efficiency in the multi-party FL while ensuring the reliability of ENs, a consortium blockchain enabled asynchronous federated learning (AFLChain) algorithm is proposed, which could dynamically allocate the learning tasks to ENs according to their computing capabilities. Moreover, an entropy weight-based reputation mechanism is introduced for the EN evaluation to further improve the performance of the AFLChain. Finally, the simulation results demonstrate the effectiveness of the proposed algorithms.
Vision Transformers with Mixed-Resolution Tokenization

Vision Transformer models process input images by dividing them into a spatially regular grid of equal-size patches. Conversely, Transformers were originally introduced over natural language sequences, where each token represents a subword – a chunk of raw data of arbitrary size. In this work, we apply this approach to Vision Transformers by introducing a novel image tokenization scheme, replacing the standard uniform grid with a mixed-resolution sequence of tokens, where each token represents a patch of arbitrary size. Using the Quadtree algorithm and a novel saliency scorer, we construct a patch mosaic where low-saliency areas of the image are processed in low resolution, routing more of the model’s capacity to important image regions. Using the same architecture as vanilla ViTs, our Quadformer models achieve substantial accuracy gains on image classification when controlling for the computational budget. Code and models are publicly available at https://github.com/TomerRonen34/mixed-resolution-vit.
Addressing the Occlusion Problem in Multi-Camera People Tracking with Human Pose Estimation

Multi-camera people tracking (MCPT) is a challenging task that is crucial for developing intelligent surveillance applications. In this work, we propose an MCPT system for Challenge Track 1 in the 2023 AI City Challenge. Specifically, we address the issue of occlusion, which causes significant changes in a person’s appearance and makes it difficult to estimate their exact location on a global map of a given area. In this paper, we present several solutions that utilize human pose estimation for overcoming this challenge. Our experimental results demonstrate that using human pose estimation significantly improves the performance of our system. Furthermore, we achieved promising results on the official evaluation set, with an IDF1 score of 86.76%. Our code is publicly available at https://github.com/nota-github/AIC2023_Track1_Nota.
Multi View Action Recognition for Distracted Driver Behavior Localization

This paper presents our approach for Track 3 (Naturalistic Driving Action Recognition) of the 2023 AI City Challenge, where the objective is to classify distracting driving activities in each untrimmed naturalistic driving video and localize the accurate temporal boundaries of them. Our solution relies on large model fine-tuning to train a base video recognition model on a small-scale video dataset. After that, we adopt multi-view multi-fold ensemble to produce fine-grained clip-level classification results. Given the recognition probabilities, a non-trivial clustering and removing post-processing algorithm is applied to generate final location proposals. Extensive experiments demonstrate that the proposed method achieves superior performance against other methods and rank the 1st on the Test-A2 of the challenge track.
Machine-Learning-Assisted Cyclostationary Spectral Analysis for Joint Signal Classification and Jammer Detection at the Physical Layer of Cognitive Radio

Cognitive radio technology was introduced as a possible solution for spectrum scarcity by exploiting dynamic spectrum access. In the last two decades, most researchers focused on enabling cognitive radios for managing the spectrum. However, due to their intelligent nature, cognitive radios can scan the radio frequency environment and change their transmission parameters accordingly on-the-fly. Such capabilities make it suitable for the design of both advanced jamming and anti-jamming systems. In this context, our work presents a novel, robust algorithm for spectrum characterisation in wideband radios. The proposed algorithm considers that a wideband spectrum is sensed by a cognitive radio terminal. The wideband is constituted of different narrowband signals that could either be licit signals or signals jammed by stealthy jammers. Cyclostationary feature detection is adopted to measure the spectral correlation density function of each narrowband signal. Then, cyclic and angular frequency profiles are obtained from the spectral correlation density function, concatenated, and used as the feature sets for the artificial neural network, which characterise each narrowband signal as a licit signal with a particular modulation scheme or a signal jammed by a specific stealthy jammer. The algorithm is tested under both multi-tone and modulated stealthy jamming attacks. Results show that the classification accuracy of our novel algorithm is superior when compared with recently proposed signal classifications and jamming detection algorithms. The applications of the algorithm can be found in both commercial and military communication systems.
Dark Web Marketplaces: Data for Collaborative Threat Intelligence

The dark web has become an increasingly important landscape for the sale of illicit cyber goods. Given the prevalence of malware and tools that are used to steal data from individuals on these markets, it is crucial that every company, governing body, and cyber professional be aware of what information is sold on these marketplaces. Knowing this information will allow these entities to protect themselves against cyber attacks and from information breaches. In this article, we announce the public release of a data set on dark web marketplaces’ cybersecurity-related listings. We spent multiple years seeking out websites that sold illicit digital goods and collected data on the available products. Due to the marketplaces’ varied and complex layers of security, we leveraged the flexible Selenium WebDriver with Python to navigate the web pages and collect data. We present analysis of categories of malicious cyber goods sold on marketplaces, prices, persistent vendors, ratings, and other basic information on marketplace storefronts. Additionally, we share the tools and techniques we’ve compiled, enabling others to scrape dark web marketplaces at a significantly lower risk. We invite professionals who opt to gather data from the dark web to contribute to the publicly shared threat intelligence resource.
Privacy‐preserving model splitting and quality‐aware device association for federated edge learning

Abstract Federated edge learning (FEEL) provides a promising device‐edge collaborative learning paradigm, which enables edge devices to parallel participate in model co‐creation while preserving user privacy, opening countless opportunities to enable edge intelligence. With the growing demand for intelligent services, extensive FEEL deployment is inevitable. Nevertheless, existing FL schemes neglect two unique features (i.e., resource heterogeneity and data heterogeneity) in real‐world edge learning and thus may negatively affect the training efficiency and accuracy. Specifically, (1) heterogeneous and limited device resources cause massive laggards, which bring intolerable training delay; (2) heterogeneous data distribution causes device quality divergence, bringing severe training accuracy degradation. This article proposes a split‐based FEEL framework and an adaptive model splitting and quality‐aware device association scheme (MSDA) to tackle the aforementioned challenges. MSDA contains two levels: at the model splitting level, according to device capability and model structure, an adaptive splitting mechanism is proposed to provide a low‐latency and privacy‐preserving model splitting strategy for each device and guide subsequent device association. At the device association level, each device is simulated as a player with a quality weight in the potential game. Then a quality‐aware decentralized device association mechanism is designed to ensure that more high‐quality devices upload local updates before the deadline with the help of the edge server. Finally, experimental results demonstrate that MSDA yields significant improvements, achieving up to 3.1 training speedup and 39% accuracy improvement compared to state‐of‐the‐art methods.