Reviewerly/exHarmony huggingface dataset:
VARS: Video Assistant Referee System for Automated Soccer Decision Making from Multiple Views

The Video Assistant Referee (VAR) has revolutionized association football, enabling referees to review incidents on the pitch, make informed decisions, and ensure fairness. However, due to the lack of referees in many countries and the high cost of the VAR infrastructure, only professional leagues can benefit from it. In this paper, we propose a Video Assistant Referee System (VARS) that can automate soccer decision-making. VARS leverages the latest findings in multi-view video analysis, to provide real-time feedback to the referee, and help them make informed decisions that can impact the outcome of a game. To validate VARS, we introduce SoccerNet-MVFoul, a novel video dataset of soccer fouls from multiple camera views, annotated with extensive foul descriptions by a professional soccer referee, and we benchmark our VARS to automatically recognize the characteristics of these fouls. We believe that VARS has the potential to revolutionize soccer refereeing and take the game to new heights of fairness and accuracy across all levels of professional and amateur federations.
Quantum Reinforcement Learning for Quantum Architecture Search

This paper presents a quantum architecture search (QAS) framework using quantum reinforcement learning (QRL) to generate quantum gate sequences for multi-qubit GHZ states. The proposed framework employs the asynchronous advantage actor-critic (A3C) algorithm to optimize the QRL agent, which has access to Pauli-X, Y, Z expectation values and a predefined set of quantum operations. Our approach does not require any prior knowledge of quantum physics. The framework can be used with other QRL architectures or optimization methods to explore gate synthesis and compilation for various quantum states.
Parameter sharing and multi-granularity feature learning for cross-modality person re-identification

Abstract Visible-infrared person re-identification aims to match pedestrian images between visible and infrared modalities, and its two main challenges are intra-modality differences and cross-modality differences between visible and infrared images. To address these issues, many advanced methods attempt to design new network structures to extract modality-sharing features, mitigate modality differences, or learn part-level features to overcome background interference. However, they ignore the parameter sharing of the convolutional layers to obtain more modality-sharing features. At the same time, only using part-level features lack discriminative pedestrian representations such as body structure and contours. To handle these problems, a parameter sharing and feature learning network is proposed in this paper to mitigate modality differences and further enhance feature discrimination. Firstly, a new two-stream parameter sharing network is proposed, by sharing the convolutional layers parameters to obtain more modality-sharing features. Secondly, a multi-granularity feature learning module is designed to reduce modality differences at both coarse and fine-grained levels while further enhancing feature discriminability. In addition, a center alignment loss is proposed to learn relationships between identities and to reduce modality differences by clustering features into their centers. For the part-level feature learning, the hetero-center triplet loss is adopted to alleviate the strict constraints of triplet loss. Finally, extensive experiments are conducted to validate our method outperforms state-of-the-art methods on two challenging datasets. In the SYSU-MM01 dataset, the Rank-1 and mAP reach $$74.0\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>74.0</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> and $$70.51\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>70.51</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> in the all-search mode, which is an increase of $$3.4\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.4</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> and $$3.61\%$$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>3.61</mml:mn> <mml:mo>%</mml:mo> </mml:mrow> </mml:math> to baseline, respectively.
The effectiveness of moderating harmful online content

In 2022, the European Union introduced the Digital Services Act (DSA), a new legislation to report and moderate harmful content from online social networks. Trusted flaggers are mandated to identify harmful content, which platforms must remove within a set delay (currently 24 h). Here, we analyze the likely effectiveness of EU-mandated mechanisms for regulating highly viral online content with short half-lives. We deploy self-exciting point processes to determine the relationship between the regulated moderation delay and the likely harm reduction achieved. We find that harm reduction is achievable for the most harmful content, even for fast-paced platforms such as Twitter. Our method estimates moderation effectiveness for a given platform and provides a rule of thumb for selecting content for investigation and flagging, managing flaggers' workload.
A Synthesis-Based Stateful Approach for Guiding Design Thinking in Embedded System Development

Embedded systems have attracted more attention and have become more critical due to the recent computer technology advancements and applications in various areas, such as healthcare, transportation, and manufacturing. Traditional software design approaches and the finite state machine cannot provide sufficient support due to two major reasons: the increasing need for more functions in designing an embedded system and sequential controls in the implementation. This deficiency particularly discourages inexperienced engineers who use conventional methods to design embedded software. Hence, we proposed a design method, the Synthesis-Based Stateful Software Design Approach (SSSDA), which synthesizes two existing methods, the Synthesis-Based Software Design Framework (SSDF) and Process and Artifact State Transition Abstraction (PASTA), to remedy the drawback of conventional methods. To show how to conduct our proposed design approach and investigate how it supports embedded system design, we studied an industrial project developed by a sophomore student team. Our results showed that our proposed approach could significantly help students lay out modules, improve testability, and reduce defects.
Formation of a Value Attitude to a Healthy Lifestyle Among Secondary School Students

In the modern world, the problem of preserving and strengthening one’s health comes first, it is one of the most important human values, a guarantee of well-being and longevity. Everyone has the opportunity to strengthen their health, preserve their ability to work, endurance, physical activity, so it is very important to start taking care of their health from an early age. Healthy lifestyle habits should be promoted from an early age in the educational process in general education institutions. A healthy lifestyle and health values are important elements of belonging to society. Those who develop a healthy lifestyle and recognize the importance of health are important members of society who contribute to the mental and physical health of children of the next generation. They are important members of society who contribute to the mental and physical health of future generations. The article presents the results of a study on the value attitude to the health of schoolchildren. The respondents showed good results, so high personal significance of health in 50% people, insufficient significance in 17% people, low significance in 33% people. The results obtained indicate that for 50% of the subjects, health is one of the most important values of life, to which they assign a high step in the hierarchy of conditions necessary for a happy life. Thus, a healthy lifestyle is considered as a value manifested in the relationship ‘person-lifestyle’ and revealed through the following ideas: the idea of the value of health, the idea of leading a healthy lifestyle, the idea of a person as a responsible subject.
Motional foreground attention-based video crowd counting

In this paper, we tackle the problem of video crowd counting. Compared with single image crowd counting, video provides gradual spatial and temporal variation information that would help to strengthen the robustness of crowd counting. Therefore, it is critical to make full use of neighboring frames both in feature extraction and final prediction for current frame’s estimation. Based on the above observations, we propose a motional foreground attention-based video crowd counting method. Specifically, we first leverage an foreground estimation module based on ConvNeXt to extract the motional features from bidirectional frame differences and output a foreground estimation map. Then the motional features combined with the static features of current frame are sent into feature fusion network, where foreground estimation map is transformed as attention weights for crowd number estimation. Three new indoor video datasets are manually annotated. The proposed method achieves state-of-the-art performance on all indoor and outdoor video datasets.
A Meta-learning Approach for Domain Generalisation across Visual Modalities in Vehicle Re-identification

Recent advances in imaging technologies have enabled the usage of infrared spectrum data for computer vision tasks previously working with traditional RGB data, such as re-identification.Infrared spectrum data can provide complementary and consistent visual information in situations of low visibility such as night-time, or adverse environments.However, the main issue that prevents the training of multi-modal systems is the lack of available infrared spectrum data.To this end, it is important to create systems that can easily adapt to data of multiple modalities, at inference time.In this paper, we propose a domain generalisation approach for multi-modal vehicle re-identification based on the recent success of meta-learning training approaches, and evaluate the ability of the model to perform to unseen modality data at testing time.In our experiments we use RGB, near-infrared and thermal-infrared modalities using the RGBNT100 dataset and prove that our meta-learning training configuration can improve the generalisation ability of the trained model compared to traditional training settings.
Preprocessing via Deep Learning for Enhancing Real-Time Performance of Object Detection

Deep learning models have significantly improved object detection essential for traffic monitoring. However, these models’ increasing complexity results in higher latency and resource consumption, making real-time object detection challenging. To address this issue, we propose a lightweight deep learning model called Empty Road Detection (ERD). ERD efficiently identifies and removes empty traffic images that do not contain any object of interest, such as vehicles, via binary classification. By serving as a preprocessing unit, ERD filters out nonessential data, reducing computational complexity and latency. ERD is highly compatible and can work seamlessly with any third-party object detection model. In our evaluation, we found that ERD improves the frame processing rate of EfficientDet, SSD, and YOLOV5 by approximately 44%, 40%, and 10%, respectively, for a real-world traffic monitoring video.
Towards Characterizing the Semantic Robustness of Face Recognition

Deep Neural Networks (DNNs) lack robustness against imperceptible perturbations to their input. Face Recognition Models (FRMs) based on DNNs inherit this vulnerability. We propose a methodology for assessing and characterizing the robustness of FRMs against semantic perturbations to their input. Our methodology causes FRMs to malfunction by designing adversarial attacks that search for identity-preserving modifications to faces. In particular, given a face, our attacks find identity-preserving variants of the face such that an FRM fails to recognize the images belonging to the same identity. We model these identity-preserving semantic modifications via direction- and magnitude-constrained perturbations in the latent space of StyleGAN. We further propose to characterize the semantic robustness of an FRM by statistically describing the perturbations that induce the FRM to malfunction. Finally, we combine our methodology with a certification technique, thus providing (i) theoretical guarantees on the performance of an FRM, and (ii) a formal description of how an FRM may model the notion of face identity.
Improving Deep Learning-based Automatic Checkout System Using Image Enhancement Techniques

The retail sector has experienced significant growth in artificial intelligence and computer vision applications, particularly with the emergence of automatic checkout (ACO) systems in stores and supermarkets. ACO systems encounter challenges such as object occlusion, motion blur, and similarity between scanned items while acquiring accurate training images for realistic checkout scenarios is difficult due to constant product updates. This paper improves existing deep learning-based ACO solutions by incorporating several image enhancement techniques in the data pre-processing step. The proposed ACO system employs a detect-and-track strategy, which involves: (1) detecting objects in areas of interest; (2) tracking objects in consecutive frames; and (3) counting objects using a track management pipeline. Several data generation techniques—including copy-and-paste, random placement, and augmentation—are employed to create diverse training data. Additionally, the proposed solution is designed as an open-ended framework that can be easily expanded to accommodate multiple tasks. The system has been evaluated on the AI City Challenge 2023 Track 4 dataset, showcasing outstanding performance by achieving a top-1 ranking on test-set A with an F1 score of 0.9792.
A Unified Multi-modal Structure for Retrieving Tracked Vehicles through Natural Language Descriptions

Through the development of multi-modal and contrastive learning, image and video retrieval have made immense progress over the last years. Organically fused text, image, and video knowledge brings huge potential opportunities for multi-dimension, and multi-view retrieval, especially in traffic senses. This paper proposes a novel Multimodal Language Vehicle Retrieval (MLVR) system, for retrieving the trajectory of tracked vehicles based on natural language descriptions. The MLVR system is mainly combined with an end-to-end text-video contrastive learning model, a CLIP few-shot domain adaption method, and a semi-centralized control optimization system. Through a comprehensive understanding the knowledge from the vehicle type, color, maneuver, and surrounding environment, the MLVR forms a robust method to recognize an effective trajectory with provided natural language descriptions. Under this structure, our approach has achieved 81.79% Mean Reciprocal Rank (MRR) accuracy on the test dataset, in the 7th AI City Challenge Track 2, Tracked-Vehicle Retrieval by Natural Language Descriptions, rendering the 2nd rank on the public leaderboard. Our code is available at https://github.com/eadst/MLVR.
NamedMask: Distilling Segmenters from Complementary Foundation Models

The goal of this work is to segment and name regions of images without access to pixel-level labels during training. To tackle this task, we construct segmenters by distilling the complementary strengths of two foundation models. The first, CLIP [26], exhibits the ability to assign names to image content but lacks an accessible representation of object structure. The second, DINO [5], captures the spatial extent of objects but has no knowledge of object names. Our method, termed NamedMask, begins by using CLIP to construct category-specific archives of images. These images are pseudo-labelled with a category-agnostic salient object detector bootstrapped from DINO, then refined by category-specific segmenters using the CLIP archive labels. Thanks to the high quality of the refined masks, we show that a standard segmentation architecture trained on these archives with appropriate data augmentation achieves impressive semantic segmentation abilities for both single-object and multi-object images. As a result, our proposed NamedMask performs favourably against a range of prior work on five benchmarks including the VOC2012, COCO and large-scale ImageNet-S datasets.
MotionTrack: End-to-End Transformer-based Multi-Object Tracking with LiDAR-Camera Fusion

Multiple Object Tracking (MOT) is crucial to autonomous vehicle perception. End-to-end transformer-based algorithms, which detect and track objects simultaneously, show great potential for the MOT task. However, most existing methods focus on image-based tracking with a single object category. In this paper, we propose an end-to-end transformer-based MOT algorithm (MotionTrack) with multi-modality sensor inputs to track objects with multiple classes. Our objective is to establish a transformer baseline for the MOT in an autonomous driving environment. The proposed algorithm consists of a transformer-based data association (DA) module and a transformer-based query enhancement module to achieve MOT and Multiple Object Detection (MOD) simultaneously. The MotionTrack and its variations achieve better results (AMOTA score at 0.55) on the nuScenes dataset compared with other classical baseline models, such as the AB3DMOT, the CenterTrack, and the probabilistic 3D Kalman filter. In addition, we prove that a modified attention mechanism can be utilized for DA to accomplish the MOT, and aggregate history features to enhance the MOD performance.
RTTLC: Video Colorization with Restored Transformer and Test-time Local Converter

Video colorization is a highly challenging and ill-posed problem that suffers from severe flickering artifacts and color distribution inconsistency. To resolve these issues, we propose a Restored Transformer and Test-time Local Converter network(RTTLC). Firstly, we introduce a Bidirectional Recurrent Block and a Learnable Guided Mask to our network. This leverages hidden knowledge from adjacent frames that include rich information about occlusion, resulting in significant enhancements in visual quality. Secondly, we integrate a Restored Transformer that enables the network to utilize more spatial contextual information and capture multi-scale information more accurately. Thirdly, during inference, we utilize the Test-time Local Converter(TLC) strategy to alleviate distribution shift and enhance the performance of the model. Experimental results show good performance of FID and CDC. Notably, RTTLC achieves second prize in both tracks of the NTIRE23 video colorization challenges.
A Real-time and Lightweight Method for Tiny Airborne Object Detection

With wide applications of unmanned aerial vehicles (UAVs), the detection of airborne objects has become crucial to ensure the flight safety of UAVs and prevent their illegal use. Although object detection has achieved great success in past years, it is still a challenging problem to detect tiny airborne objects. To solve this problem, we propose a simple and effective Tiny Airborne object Detection (TAD) method. It locates potential objects using inconsistent motion cues between airborne objects and backgrounds instead of the low-quality representation of tiny objects. This enables TAD to sensitively detect tiny objects with limited appearance information. Specifically, we first establish correspondences of pixels between adjacent frames based on the local similarity of spatial feature vectors to achieve motion modeling. Next, the local similarity of motion patterns is computed to explicitly describe the motion consistency of each position with its surrounding pixels. Then, a simple network is used to output the heatmap that reflects the probability of object presence. A higher probability of containing an object will be assigned to positions with a greater difference in motion from their surrounding pixels. Finally, an independent network branch is employed to regress center offsets and scale information of objects, which are used to correct the error in the estimated object position from the heatmap and obtain the final bounding box, respectively. Experiments on three challenging datasets demonstrate that the proposed method can achieve advanced performance. Notably, TAD is highly lightweight, and the detection speed is significantly better than existing methods.
Video Quality Assessment Based on Swin Transformer with Spatio-Temporal Feature Fusion and Data Augmentation

While video enhancement has drawn significant interest and has been extensively studied by academia and industry, the corresponding research on video quality assessment (VQA) for enhanced video has not been widely addressed. Video enhancement methods normally change the relevant metrics like brightness, contrast, color, etc., leading to the fluctuation of perceptual quality and challenging the related VQA task. In this paper, we propose a novel approach for VQA task based on Swin Transformer with improved spatio-temporal feature fusion, which precisely mines the stage-wise feature concatenation and provides competitive assessment performance. In addition, we propose an efficient data augmentation strategy to improve data diversity and further enhance assessment accuracy. Experimental results demonstrate that the proposed approach achieves state-of-the-art performance on two benchmark VQA datasets, and ranks first in CVPR NTIRE 2023 Quality Assessment for Video Enhancement Challenge, which proves that the proposed approach is not only promising in VQA for enhanced video but also ubiquitous in general VQA tasks.
Temporal Consistent Automatic Video Colorization via Semantic Correspondence

Video colorization task has recently attracted wide attention. Recent methods mainly work on the temporal consistency in adjacent frames or frames with small interval. However, it still faces severe challenge of the inconsistency between frames with large interval. To address this issue, we propose a novel video colorization framework, which combines semantic correspondence into automatic video colorization to keep long-range consistency. Firstly, a reference colorization network is designed to automatically colorize the first frame of each video, obtaining a reference image to supervise the following whole colorization process. Such automatically colorized reference image can not only avoid labor-intensive and time-consuming manual selection, but also enhance the similarity between reference and grayscale images. Afterwards, a semantic correspondence network and an image colorization network are introduced to colorize a series of the remaining frames with the help of the reference. Each frame is supervised by both the reference image and the immediately colorized preceding frame to improve both short-range and long-range temporal consistency. Extensive experiments demonstrate that our method outperforms other methods in maintaining temporal consistency both qualitatively and quantitatively. In the NTIRE 2023 Video Colorization Challenge, our method ranks at the 3rd place in Color Distribution Consistency (CDC) Optimization track. Code will be available online at https://github.com/bupt-ai-cz/TCVC