Reviewerly/exHarmony huggingface dataset:
Explaining Vision and Language through Graphs of Events in Space and Time

Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.
Towards Smart Education in the Industry 5.0 Era: A Mini Review on the Application of Federated Learning

The 5.0 era's arrival and the ongoing advancement of technology have had a significant impact on many facets of our society, including education. There is increased interest in employing federated learning (FL) strategies to enhance educational processes as a result of the expansion of educational data and the accessibility of digital platforms. FL proves to be a promising approach to leverage distributed data sources while preserving privacy and data security. This paper attempts to offer a thorough review of the most recent FL research projects and applications in the education sector during the 5.0 era. We examine the fundamental concepts of FL and explore its potential applications in the context of education namely in terms of providing personalised education, transforming the education landscape, achieving student outcomes, ensuring student engagement, preserving security and privacy implications for education, and related adaptive assessments. Additionally, we also discuss the challenges and considerations associated with implementing FL in educational settings. The paper will thus serve as a valuable resource for researchers, educators, policymakers, and industry professionals who are interested in leveraging FL to unlock the complete potential of educational data while ensuring privacy and security in the dynamic 5.0 era.
Fine-Grained is Too Coarse: A Novel Data-Centric Approach for Efficient Scene Graph Generation

Learning to compose visual relationships from raw images in the form of scene graphs is a highly challenging task due to contextual dependencies, but it is essential in computer vision applications that depend on scene understanding. However, no current approaches in Scene Graph Generation (SGG) aim at providing useful graphs for downstream tasks. Instead, the main focus has primarily been on the task of unbiasing the data distribution for predicting more fine-grained relations. That being said, all fine-grained relations are not equally relevant and at least a part of them are of no use for real-world applications. In this work, we introduce the task of Efficient SGG that prioritizes the generation of relevant relations, facilitating the use of Scene Graphs in downstream tasks such as Image Generation. To support further approaches, we present a new dataset, VG150-curated, based on the annotations of the popular Visual Genome dataset. We show through a set of experiments that this dataset contains more high-quality and diverse annotations than the one usually use in SGG. Finally, we show the efficiency of this dataset in the task of Image Generation from Scene Graphs <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Confusion Mixup Regularized Multimodal Fusion Network for Continual Egocentric Activity Recognition

Continual egocentric activity recognition aims to understand diverse first-person activities from the multimodal data of a wearable device captured in streaming environments, which is an emerging and challenging task. Existing continual learning methods ignore the dynamic change of multiple modalities’ correlation and hardly learn discriminative representations for the sequentially isolated activity classes from different stages. In this paper, we propose a Confusion Mixup Regularized Multimodal Fusion Network (CMR-MFN) to address this issue. Firstly, CMR-MFN is composed of a ternary-modality-input dynamic expansion architecture, which progressively grows additional branches for in-stage class recognition. Each input owns a frozen modality-specific backbone to avoid forgetting caused by parameter shifts. Secondly, CMR-MFN captures the dynamics of multimodal inputs via learnable self-attention layers. We augment unknown classes by linearly mixing up the samples from two known classes and assigning a biased weight to one of them, which makes the unknown class samples confusing toward the known class with a higher weight. By learning from the current and augmented training data together, we regularize the multimodal fusion representation to distinguish the in-stage classes from their confusing samples of unknown classes, which implicitly pushes the out-stage classes’ samples far from the in-stage classes’ ones when they are similar to each other. Experiments show that the proposed method significantly outperforms state-of-the-art methods for multimodal continual egocentric activity recognition. Our code is available at https://github.com/Hanna-W/CMR-MFN.
Using a Tuned Transfer Learning Model to Identify and Localize White Blood Cells

White Blood Cells (WBC) are one of the most significant blood cell components. These cells guard against infection-causing organisms like bacteria, viruses, and fungi. There are five distinct types of white blood cells in the human body. These cells have numerous names, including lymphocytes, monocytes, eosinophils, basophils, and neutrophils. An abnormally high or low white blood cell count might result in various diseases. A dataset of 12,500 JPEG images is divided into four WBC classes. These are Eosinophil, Lymphocyte, Monocyte, and Neutrophil classes. Each class contains about 3000 images. The approximate color of the WBC has been determined by selecting a series of images and analyzing the region of interest. The extracted color was employed as a centroid to segment the image using clustering algorithms. The Region of Interest (RoI) bounding boxes have been trimmed to create a new segmented dataset. CNN's Resnet 50 model was utilized for training and obtaining the results. The dataset has been successively divided into 80:20 train: test sets. After 25 epochs, the model's loss = 6.2305e-04, accuracy = 0.98, validation loss = 0.13, validation accuracy = 0.96, and learning rate = 1.0000e-06 were determined. The model can detect, identify WBC types and localize them.
Sharing is Caring: Concurrent Interactive Segmentation and Model Training using a Joint Model

The performance of neural predictors depends on the size and composition of the training dataset. However, annotating data is expensive. Efficient annotation systems usually feature a neural annotation predictor whose result can be edited by the expert using classical tools. Existing systems train the annotation predictor from an initial small subset of data annotated by classical tools and then freeze it for the rest of the annotation process. This is suboptimal as the annotation predictor does not benefit from the new annotations as the annotation process progresses. We propose a framework called Single Active Interactive Model (SAIM), which integrates the three steps of data selection, annotation and training into a single architecture. This is made possible by three key properties of SAIM in contrast with existing work: 1) SAIM uses a deep interactive predictor; hence the classical tools are not required and the annotation predictor can be pre-trained with limited data to produce quality annotations; 2) SAIM uses a single model shared between the three steps, hence the model is deployable and the annotation predictor improves as annotation progresses; 3) SAIM uses active learning to maximise the impact of each annotation on the predictor performance, making the model rapidly improve. We evaluated SAIM by emulating annotation scenarios on fully-labelled segmentation datasets. For a complex female pelvis MRI dataset, pre-training SAIM on 15% of data and annotating the whole dataset achieves 73.4% IoU with 6.3 hours of annotation time, against 75.8% IoU for complete manual annotation, requiring 40.0 hours. We also applied SAIM to a real-world case of very large MRI dataset (AMOS) segmentation, which cannot be feasibly annotated otherwise.
The Application of Infrared Thermal Imaging Technology in State Detection of Porcelain Insulators

The principles, advantages and disadvantages of various detection methods for porcelain insulators are compared and analyzed. The methods and requirements of using infrared thermal image to detect the running state of porcelain insulators are discussed. The faulty and low value insulators are identified according to the abnormal conditions of the infrared atlas and temperature distribution curves of one porcelain insulators. And the laboratory test results verified that the infrared thermal image method can identify the faulty insulator accurately and quickly. Finally, the validity and accuracy of infrared thermal image method are further verified by comparison with the detection results of the short circuit fork method, and the measures and suggestions for improving the safe operation of porcelain insulator are given.
The First Visual Object Tracking Segmentation VOTS2023 Challenge Results

The Visual Object Tracking Segmentation VOTS2023 challenge is the eleventh annual tracker benchmarking activity of the VOT initiative. This challenge is the first to merge short-term and long-term as well as single-target and multiple-target tracking with segmentation masks as the only target location specification. A new dataset was created; the ground truth has been withheld to prevent overfitting. New performance measures and evaluation protocols have been created along with a new toolkit and an evaluation server. Results of the presented 47 trackers indicate that modern tracking frameworks are well-suited to deal with convergence of short-term and long-term tracking and that multiple and single target tracking can be considered a single problem. A leaderboard, with participating trackers details, the source code, the datasets, and the evaluation kit are publicly available at the challenge website <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
An Experimental Protocol for Neural Architecture Search in Super-Resolution

Neural architecture search has seen continual progress due to the interest in automating architecture design in deep learning following the promise of finding the best possible neural network architecture tailored for a particular task. Recently, many works focused on tackling tasks like image classification and language modeling, allowing significant developments in computer vision and NLP. As research in such directions has established standard criteria and benchmarking tasks for algorithmic performance comparison, the same cannot be said of other applications and tasks. Our work presents an experimental comparison protocol that narrows down the process of evaluating super-resolution image restoration architectures in neural architecture search approaches. Such protocol consists of two datasets for training and validation during and after the architecture search, and the application of a Bayesian statistical test for studying the observable results.
Context-VQA: Towards Context-Aware and Purposeful Visual Question Answering

Visual question answering (VQA) has the potential to make the Internet more accessible in an interactive way, allowing people who cannot see images to ask questions about them. However, multiple studies have shown that people who are blind or have low-vision prefer image explanations that incorporate the context in which an image appears, yet current VQA datasets focus on images in isolation. We argue that VQA models will not fully succeed at meeting people’s needs unless they take context into account. To further motivate and analyze the distinction between different contexts, we introduce Context-VQA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , a VQA dataset that pairs images with contexts, specifically types of websites (e.g., a shopping website). We find that the types of questions vary systematically across contexts. For example, images presented in a travel context garner 2 times more "Where?" questions, and images on social media and news garner 2.8 and 1.8 times more "Who?" questions than the average. We also find that context effects are especially important when participants can’t see the image. These results demonstrate that context affects the types of questions asked and that VQA models should be context-sensitive to better meet people’s needs, especially in accessibility settings.
Probabilistic MIMO U-Net: Efficient and Accurate Uncertainty Estimation for Pixel-wise Regression

Uncertainty estimation in machine learning is paramount for enhancing the reliability and interpretability of predictive models, especially in high-stakes real-world scenarios. Despite the availability of numerous methods, they often pose a trade-off between the quality of uncertainty estimation and computational efficiency. Addressing this challenge, we present an adaptation of the Multiple-Input Multiple-Output (MIMO) framework - an approach exploiting the overparameterization of deep neural networks - for pixel-wise regression tasks. Our MIMO variant expands the applicability of the approach from simple image classification to broader computer vision domains. For that purpose, we adapted the U-Net architecture to train multiple subnetworks within a single model, harnessing the overparameterization in deep neural networks. Additionally, we introduce a novel procedure for synchronizing subnetwork performance within the MIMO framework. Our comprehensive evaluations of the resulting MIMO U-Net on two orthogonal datasets demonstrate comparable accuracy to existing models, superior calibration on in-distribution data, robust out-of-distribution detection capabilities, and considerable improvements in parameter size and inference time. Code available at github.com/antonbaumann/MIMO-Unet.
NOVA: NOvel View Augmentation for Neural Composition of Dynamic Objects

We propose a novel-view augmentation (NOVA) strategy to train NeRFs for photo-realistic 3D composition of dynamic objects in a static scene. Compared to prior work, our framework significantly reduces blending artifacts when inserting multiple dynamic objects into a 3D scene at novel views and times; achieves comparable PSNR without the need for additional ground truth modalities like optical flow; and overall provides ease, flexibility, and scalability in neural composition. Our codebase is on GitHub.
Real Time Practical Face Recognition System of Distorted Images Using PCA

Different homes and businesses require biometric identification, and security is one of the most important considerations. This paper aims to show how users may utilize face detection and recognition to improve the security of critical site entrances. The face is a multi-dimensional, complicated structure that requires advanced computers to detect and recognize. Face detection, identification, and automatic door access control are the three sections of this study. The Viola-Jones technique is used to detect the face, and the Principal Component Analysis (PCA) methodology is used to recognize it. The PCA method was chosen because of its quick identification time, low noise sensitivity, and ability to decrease facial picture dimensions without sacrificing key characteristics. There has been the development of a security system. After Matlab processing, HC-12 uses Arduino UNO (send and receive) controllers. A built-in camera is used to capture the individual's image first. The recorded image is then sent to a database created in MATLAB. The recorded image is compared to the training image in the database to determine the particular instance. Suppose the system detects the person as an authorized user. A signal is delivered to the Arduino UNO microcontroller via an HC-12 receiver as the former opens the gate or the door.
Adversarial Keyword Extraction and Semantic-Spatial Feature Aggregation for Clinical Report Guided Thyroid Nodule Segmentation

Existing thyroid nodule segmentation methods are primarily developed based on ultrasound images, which generally neglects the clinical reports that include rich semantic information for nodules. However, current text guided segmentation methods for natural images are not applicable to the image-report thyroid nodule dataset, due to the many-to-one correspondence between images and reports in current data. To this end, we propose a clinical report guided thyroid nodule segmentation framework with Adversarial Keyword Extraction (AKE) module to extract keywords from reports and Semantic-Spatial Feature Aggregation (SSFA) module to integrate reports into the segmentation model. To alleviate the many-to-one correspondence issue, we devise the AKE module to highlight the keywords about current ultrasound images from clinical reports with a keywords mask, which adopts adversarial learning to encourage the mask generator to mask out the useful descriptions to boost segmentation performance. We further propose the SSFA module to effectively and efficiently map semantic information from reports to each pixel of spatial features, so as to emphasize the target regions. Moreover, we manually collect a clinical Reports Assisted Thyroid Nodule segmentation dataset (RATN), which includes the ultrasound images, the pixel-wise nodule segmentation annotation, and the clinical reports. Extensive experiments have been conducted on the RATN dataset, and the results prove the effectiveness and computational efficiency of the proposed method over the existing methods. Code and data are available at https://github.com/cvi-szu .
LORD: Leveraging Open-Set Recognition with Unknown Data

Handling entirely unknown data is a challenge for any deployed classifier. Classification models are typically trained on a static pre-defined dataset and are kept in the dark for the open unassigned feature space. As a result, they struggle to deal with out-of-distribution data during inference. Addressing this task on the class-level is termed open-set recognition (OSR). However, most OSR methods are inherently limited, as they train closed-set classifiers and only adapt the downstream predictions to OSR.This work presents LORD, a framework to Leverage Open-set Recognition by exploiting unknown Data. LORD explicitly models open space during classifier training and provides a systematic evaluation for such approaches. We identify three model-agnostic training strategies that exploit background data and applied them to well-established classifiers. Due to LORD’s extensive evaluation protocol, we consistently demonstrate improved recognition of unknown data. The benchmarks facilitate in-depth analysis across various requirement levels. To mitigate dependency on extensive and costly background datasets, we explore mixup as an off-the-shelf data generation technique. Our experiments highlight mixup’s effectiveness as a substitute for background datasets. Lightweight constraints on mixup synthesis further improve OSR performance.
Bi-Encoder Cascades for Efficient Image Search

Modern neural encoders offer unprecedented text-image retrieval (TIR) accuracy, but their high computational cost impedes an adoption to large-scale image searches. To lower this cost, model cascades use an expensive encoder to refine the ranking of a cheap encoder. However, existing cascading algorithms focus on cross-encoders, which jointly process text-image pairs, but do not consider cascades of bi-encoders, which separately process texts and images. We introduce the small-world search scenario as a realistic setting where bi-encoder cascades can reduce costs. We then propose a cascading algorithm that leverages the small-world search scenario to reduce lifetime image encoding costs of a TIR system. Our experiments show cost reductions by up to 6x.
Calibrated Out-of-Distribution Detection with a Generic Representation

Out-of-distribution detection is a common issue in deploying vision models in practice and solving it is an essential building block in safety critical applications. Most of the existing OOD detection solutions focus on improving the OOD robustness of a classification model trained exclusively on in-distribution (ID) data. In this work, we take a different approach and propose to leverage generic pre-trained representation. We propose a novel OOD method, called GROOD, that formulates the OOD detection as a Neyman-Pearson task with well calibrated scores and which achieves excellent performance, predicated by the use of a good generic representation. Only a trivial training process is required for adapting GROOD to a particular problem. The method is simple, general, efficient, calibrated and with only a few hyper-parameters. The method achieves state-of-the-art performance on a number of OOD benchmarks, reaching near perfect performance on several of them. The source code is available at https://github.com/vojirt/GROOD.
JEDI: Joint Expert Distillation in a Semi-Supervised Multi-Dataset Student-Teacher Scenario for Video Action Recognition

We propose JEDI, a multi-dataset semi-supervised learning method, which efficiently combines knowledge from multiple experts, learned on different datasets, to train and improve the performance of individual, per dataset, student models. Our approach achieves this by addressing two important problems in current machine learning research: generalization across datasets and limitations of supervised training due to scarcity of labeled data. We start with an arbitrary number of experts, pretrained on their own specific dataset, which form the initial set of student models. The teachers are immediately derived by concatenating the feature representations from the penultimate layers of the students. We then train all models in a student-teacher semi-supervised learning scenario until convergence. In our efficient approach, student-teacher training is carried out jointly and end-to-end, showing that both students and teachers improve their generalization capacity during training. We validate our approach on four video action recognition datasets. By simultaneously considering all datasets within a unified semi-supervised setting, we demonstrate significant improvements over the initial experts.