Reviewerly/exHarmony huggingface dataset:
Advancing Visual Grounding with Scene Knowledge: Benchmark and Method

Visual grounding (VG) aims to establish fine-grained alignment between vision and language. Ideally, it can be a testbed for vision-and-language models to evaluate their understanding of the images and texts and their reasoning abilities over their joint space. However, most existing VG datasets are constructed using simple description texts, which do not require sufficient reasoning over the images and texts. This has been demonstrated in a recent study [27], where a simple LSTM-based text encoder without pretraining can achieve state-of-the-art performance on mainstream VG datasets. Therefore, in this paper, we propose a novel benchmark of Scene Knowledge-guided Visual Grounding (SK-VG), where the image content and referring expressions are not sufficient to ground the target objects, forcing the models to have a reasoning ability on the long-form scene knowledge. To perform this task, we propose two approaches to accept the triple-type input, where the former embeds knowledge into the image features before the image-query interaction; the latter leverages linguistic structure to assist in computing the image-text matching. We conduct extensive experiments to analyze the above methods and show that the proposed approaches achieve promising results but still leave room for improvement, including performance and interpretability. The dataset and code are available at https://github.com/zhjohnchan/SK-VG.
A Multimodal Spatio-temporal Model for Micro-Video Emotion Classification

Spatio-temporal features are information unique to video compared to images, and are an important source of features for video content analysis. The introduction of spatio-temporal features has brought a significant change to affective computing, which is no longer limited to single-image and ignores temporal affective changes. However, existing affective computing models still do not introduce spatio-temporal features. For example, facial emotion classification still uses single-image as input to the model and ignores spatio-temporal features. Therefore, we propose a novel neural network that takes video as the input of the model and extracts feature information containing spatio-temporal features in it. The extracted feature information consists of two modalities, which are audio spatio-temporal feature information and image spatio-temporal feature information. Decision-level fusion of the two modal information is performed to achieve emotion classification. Where a feature extractor based on VGG implementation is used as the audio emotion feature extraction module, which extracts the spectrogram feature information while focusing on its temporal feature information. And using the feature extractor implemented based on I3D as the image emotion feature extraction module, which consists of 3D convolutional blocks. Compared with 2D convolutional blocks, it not only focuses on image features, but also performs convolutional extraction for both spatial features and temporal features of the images within the blocks to obtain image features containing spatio-temporal features. By fusing multiple spatio-temporal feature classifiers in the best way, our model achieves an accuracy of 62.41%. The experimental results show that feature information containing spatio-temporal features is more effective in sentiment classifiers than feature information containing only image features.
Secure Crowdsourced Blockchain Computation Intelligence for IoT Systems

With the rapid development of scientific and technological revolution, the Internet of Thing systems have been widely deployed in many vertical domains such as smart cities, smart grids, smart agriculture, smart medical healthcare, etc. In these scenarios, it is an urgent problem to provide a high-effect IoT solution to support high computation ability with strong data security and privacy constraints. In this paper, we propose a novel secure data computing platform based on blockchain crowdsourcing computation. Each data sample is transmitted to its private blockchain for secure data storage. To support a high computation ability, we present a powerful edge network constituted of nearby edge gateways, edge devices, etc. Thus, these data on the blockchain can be partitioned and allocated to these edge nodes in a crowdsourcing way. We also deploy different AI tools to meet the requirements of vertical applications. Finally, we evaluate our system performance with a real crowdsourced blockchain computation platform to show our effectiveness in vertical application, e.g., face detection.
Grounding Counterfactual Explanation of Image Classifiers to Textual Concept Space

Concept-based explanation aims to provide concise and human-understandable explanations of an image classifier. However, existing concept-based explanation methods typically require a significant amount of manually collected concept-annotated images. This is costly and runs the risk of human biases being involved in the explanation. In this paper, we propose Counterfactual explanation with text-driven concepts (CounTEX), where the concepts are defined only from text by leveraging a pretrained multimodal joint embedding space without additional concept-annotated datasets. A conceptual counterfactual explanation is generated with text-driven concepts. To utilize the text-driven concepts defined in the joint embedding space to interpret target classifier outcome, we present a novel projection scheme for mapping the two spaces with a simple yet effective implementation. We show that CounTEX generates faithful explanations that provide a semantic understanding of model decision rationale robust to human bias.
SiamTDR: Time-Efficient RGBT Tracking via Disentangled Representations

The growing demand for vision tasks utilizing RGB-T (Red-Green-Blue-Thermal) imagery is attributed to the advantageous synergistic effect of combining RGB images with thermal (Tir) image information. Due to their exceptional real-time inference efficacy, siamese networks have garnered considerable attention in RGB-T object tracking as a leading solution. However, current RGB-T Siamese trackers still need to catch up with online training RGB-T trackers regarding accuracy and robustness due to ineffective utilization of valid information from both modes. To this end, this work proposes SiamTDR, a high-speed Siamese network-based RGB-T tracker with a disentangled representation and deconstructed features. Firstly, we introduce a single-modal feature extraction network into the Siamese network to capture cross-level information within unimodal features extracted from RGB or Tir images. Next, we employ a disentangled representation multi-modal feature fusion module (DP-MF) to extract cross-modal information between RGB and thermal features, thereby improving the information utilization of both modalities. Finally, a dual branch fusion module (DBF) significantly enhances the robustness of our tracker in the final bounding box selection stage. Besides, we also employ data augmentation techniques such as central random offset. Extensive experiments conducted on two RGB-T tracking benchmark datasets demonstrate the superior performance of our method, which achieves a tracking speed of over 127 frames per second (FPS) on the GTOT dataset.
Shape-Aware Text-Driven Layered Video Editing

Temporal consistency is essential for video editing applications. Existing work on layered representation of videos allows propagating edits consistently to each frame. These methods, however, can only edit object appearance rather than object shape changes due to the limitation of using a fixed UV mapping field for texture atlas. We present a shape-aware, text-driven video editing method to tackle this challenge. To handle shape changes in video editing, we first propagate the deformation field between the input and edited keyframe to all frames. We then leverage a pre-trained text-conditioned diffusion model as guidance for refining shape distortion and completing unseen regions. The experimental results demonstrate that our method can achieve shape-aware consistent video editing and compare favorably with the state-of-the-art.
Interactive Cartoonization with Controllable Perceptual Factors

Cartoonization is a task that renders natural photos into cartoon styles. Previous deep cartoonization methods only have focused on end-to-end translation, which may hinder editability. Instead, we propose a novel solution with editing features of texture and color based on the cartoon creation process. To do that, we design a model architecture to have separate decoders, texture and color, to decouple these attributes. In the texture decoder, we propose a texture controller, which enables a user to control stroke style and abstraction to generate diverse cartoon textures. We also introduce an HSV color augmentation to induce the networks to generate diverse and controllable color translation. To the best of our knowledge, our work is the first deep approach to control the cartoonization at inference while showing profound quality improvement over to baselines.
Mathematical Elements of Elementary Education

Abstract In recent decades, several Russian schools have been implementing a world-unique education program of mathematics for elementary schools. In it, the landscape of school arithmetic is radically expanded due to the basic objects of modern mathematics and computer science. These objects and their operations are visual, making them much more comprehensible than traditional arithmetic. The range of activities also expands due to, for example, the introduction of strategies for enumeration, game winning, and algorithms (also operating in a visual environment). At the same time, the student’s position changes: they independently discover and build mathematics and constantly solve personally new, but feasible tasks that are “not-known-how-to-solve.” The student’s resources are saved by using a computer to perform routine arithmetic operations that have already been discovered and understood. The implementation of this approach is discussed in detail and is illustrated by examples of actual tasks that are representative of the program under consideration and the whole approach.
AttentionShift: Iteratively Estimated Part-Based Attention Map for Pointly Supervised Instance Segmentation

Pointly supervised instance segmentation (PSIS) learns to segment objects using a single point within the object extent as supervision. Challenged by the non-negligible semantic variance between object parts, however, the single supervision point causes semantic bias and false segmentation. In this study, we propose an AttentionShift method, to solve the semantic bias issue by iteratively decomposing the instance attention map to parts and estimating fine-grained semantics of each part. AttentionShift consists of two modules plugged on the vision transformer backbone: (i) token querying for pointly supervised attention map generation, and (ii) key-point shift, which re-estimates part-based attention maps by key-point filtering in the feature space. These two steps are iteratively performed so that the part-based attention maps are optimized spatially as well as in the feature space to cover full object extent. Experiments on PASCAL VOC and MS COCO 2017 datasets show that AttentionShift respectively improves the state-of-the-art of by 7.7% and 4.8% under mAP@0.5, setting a solid PSIS baseline using vision transformer.
Complementary Intrinsics from Neural Radiance Fields and CNNs for Outdoor Scene Relighting

Relighting an outdoor scene is challenging due to the diverse illuminations and salient cast shadows. Intrinsic image decomposition on outdoor photo collections could partly solve this problem by weakly supervised labels with albedo and normal consistency from multiview stereo. With neural radiance fields (NeRF), editing the appearance code could produce more realistic results without interpreting the outdoor scene image formation explicitly. This paper proposes to complement the intrinsic estimation from volume rendering using NeRF and from inversing the photometric image formation model using convolutional neural networks (CNNs). The former produces richer and more reliable pseudo labels (cast shadows and sky appearances in addition to albedo and normal) for training the latter to predict interpretable and editable lighting parameters via a single-image prediction pipeline. We demonstrate the advantages of our method for both intrinsic image decomposition and relighting for various real outdoor scenes.
Pose Synchronization under Multiple Pair-wise Relative Poses

Pose synchronization, which seeks to estimate consistent absolute poses among a collection of objects from noisy relative poses estimated between pairs of objects in isolation, is a fundamental problem in many inverse applications. This paper studies an extreme setting where multiple relative pose estimates exist between each object pair, and the majority is incorrect. Popular methods that solve pose synchronization via recovering a low-rank matrix that encodes relative poses in block fail under this extreme setting. We introduce a three-step algorithm for pose synchronization under multiple relative pose inputs. The first step performs diffusion and clustering to compute the candidate poses of the input objects. We present a theoretical result to justify our diffusion formulation. The second step jointly optimizes the best pose for each object. The final step refines the output of the second step. Experimental results on benchmark datasets of structure-from-motion and scan-based geometry reconstruction show that our approach offers more accurate absolute poses than state-of-the-art pose synchronization techniques.
BiFormer: Learning Bilateral Motion Estimation via Bilateral Transformer for 4K Video Frame Interpolation

A novel 4K video frame interpolator based on bilateral transformer (BiFormer) is proposed in this paper, which performs three steps: global motion estimation, local motion refinement, and frame synthesis. First, in global motion estimation, we predict symmetric bilateral motion fields at a coarse scale. To this end, we propose BiFormer, the first transformer-based bilateral motion estimator. Second, we refine the global motion fields efficiently using blockwise bilateral cost volumes (BBCVs). Third, we warp the input frames using the refined motion fields and blend them to synthesize an intermediate frame. Extensive experiments demonstrate that the proposed BiFormer algorithm achieves excellent interpolation performance on 4K datasets. The source codes are available at https://github.com/JunHeum/BiFormer.
TarViS: A Unified Approach for Target-Based Video Segmentation

The general domain of video segmentation is currently fragmented into different tasks spanning multiple benchmarks. Despite rapid progress in the state-of-the-art, current methods are overwhelmingly task-specific and cannot conceptually generalize to other tasks. Inspired by recent approaches with multi-task capability, we propose TarViS: a novel, unified network architecture that can be applied to any task that requires segmenting a set of arbitrarily defined ‘targets' in video. Our approach is flexible with respect to how tasks define these targets, since it models the latter as abstract ‘queries' which are then used to predict pixel-precise target masks. A single TarViS model can be trained jointly on a collection of datasets spanning different tasks, and can hot-swap between tasks during inference without any task-specific retraining. To demonstrate its effectiveness, we apply TarViS to four different tasks, namely Video Instance Segmentation (VIS), Video Panoptic Segmentation (VPS), Video Object Segmentation (VOS) and Point Exemplar-guided Tracking (PET). Our unified, jointly trained model achieves state-of-the-art performance on 5/7 benchmarks spanning these four tasks, and competitive performance on the remaining two. Code and model weights are available at: https://github.com/Ali2500/TarViS
Revisiting Self-Similarity: Structural Embedding for Image Retrieval

Despite advances in global image representation, existing image retrieval approaches rarely consider geometric structure during the global retrieval stage. In this work, we revisit the conventional self-similarity descriptor from a convolutional perspective, to encode both the visual and structural cues of the image to global image representation. Our proposed network, named Structural Embedding Network (SENet), captures the internal structure of the images and gradually compresses them into dense self-similarity descriptors while learning diverse structures from various images. These self-similarity descriptors and original image features are fused and then pooled into global embedding, so that global embedding can represent both geometric and visual cues of the image. Along with this novel structural embedding, our proposed network sets new state-of-the-art performances on several image retrieval benchmarks, convincing its robustness to look-alike distractors. The code and models are available: https://github.com/sungonce/SENet.
Path Planning for Amphibious Robots based on Multi-optimization Strategy A* Algorithm

For wheel-propeller amphibious robot, finding the optimal motion path is essential to improve the efficiency of autonomous navigation, as they have energy limitations in completing tasks such as detecting and autonomously recovering photovoltaic equipment on the water surface. The A* algorithm is widely used for the autonomous navigation of robots, but traditional A* algorithms have problems such as too many turning points, low efficiency, and proximity to obstacles. To address these issues, this paper proposes a multi-optimization strategy A* algorithm aimed at improving the efficiency and safety of the robot’s autonomous navigation. This paper compares the algorithm before and after optimization by simulation, and the results show that the optimized algorithm has solved the deficiencies of traditional A* algorithm, and has fast speed and safety.
FAIR Sensor Ecosystem: Long-Term (Re-)Usability of FAIR Sensor Data through Contextualization

The long-term utility and reusability of measurement data from production processes depend on the appropriate contextualization of the measured values. These requirements further mandate that modifications to the context need to be recorded. To be (re-)used at all, the data must be easily findable in the first place, which requires arbitrary filtering and searching routines. Following the FAIR guiding principles, fostering findable, accessible, interoperable and reusable (FAIR) data, in this paper, the FAIR Sensor Ecosystem is proposed, which provides a contextualization middleware based on a unified data metamodel. All information and relations which might change over time are versioned and associated with temporal validity intervals to enable full reconstruction of a system’s state at any point in time. A technical validation demonstrates the correctness of the FAIR Sensor Ecosystem, including its contextualization model and filtering techniques. State-of-the-art FAIRness assessment frameworks rate the proposed FAIR Sensor Ecosystem with an average FAIRness of 71%. The obtained rating can be considered remarkable, as deductions mainly result from the lack of fully appropriate FAIRness metrics and the absence of relevant community standards for the domain of the manufacturing industry.
Hierarchical Neural Memory Network for Low Latency Event Processing

This paper proposes a low latency neural network architecture for event-based dense prediction tasks. Conventional architectures encode entire scene contents at a fixed rate regardless of their temporal characteristics. Instead, the proposed network encodes contents at a proper temporal scale depending on its movement speed. We achieve this by constructing temporal hierarchy using stacked latent memories that operate at different rates. Given low latency event steams, the multi-level memories gradually extract dynamic to static scene contents by propagating information from the fast to the slow memory modules. The architecture not only reduces the redundancy of conventional architectures but also exploits long-term dependencies. Further-more, an attention-based event representation efficiently encodes sparse event streams into the memory cells. We conduct extensive evaluations on three event-based dense prediction tasks, where the proposed approach outperforms the existing methods on accuracy and latency, while demonstrating effective event and image fusion capabilities. The code is available at https://hamarh.github.io/hmnet/.
Common Pets in 3D: Dynamic New-View Synthesis of Real-Life Deformable Categories

Obtaining photorealistic reconstructions of objects from sparse views is inherently ambiguous and can only be achieved by learning suitable reconstruction priors. Earlier works on sparse rigid object reconstruction successfully learned such priors from large datasets such as CO3D. In this paper, we extend this approach to dynamic objects. We use cats and dogs as a representative example and introduce Common Pets in 3D (CoP3D), a collection of crowd-sourced videos of approximately 4,200 distinct pets. CoP3D is one of the first large-scale datasets for benchmarking non-rigid 3D reconstruction “in the wild”. We also propose Tracker-NeRF, a method for learning 4D reconstruction from our dataset. At test time, given a small number of video frames of an unseen object, Tracker-NeRF predicts the trajectories of its 3D points and generates new views, interpolating viewpoint and time. Results on CoP3D reveal significantly better non-rigid new-view synthesis performance than existing baselines. The data will be available on the project webpage: https://cop3d.github.io/.
HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models

Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions. Recently, Contrastive Language-Image Pre-training (CLIP) has shown great potential in providing interaction prior for HOI detectors via knowledge distillation. However, such approaches often rely on large-scale training data and suffer from inferior performance under few/zero-shot scenarios. In this paper, we propose a novel HOI detection framework that efficiently extracts prior knowledge from CLIP and achieves better generalization. In detail, we first introduce a novel interaction decoder to extract informative regions in the visual feature map of CLIP via a cross-attention mechanism, which is then fused with the detection backbone by a knowledge integration block for more accurate human- object pair detection. In addition, prior knowledge in CLIP text encoder is leveraged to generate a classifier by embedding HOI descriptions. To distinguish fine-grained interactions, we build a verb classifier from training data via visual semantic arithmetic and a lightweight verb representation adapter. Furthermore, we propose a training-free enhancement to exploit global HOI predictions from CLIP. Extensive experiments demonstrate that our method outperforms the state of the art by a large margin on various settings, e.g. +4.04 mAP on HICO-Det. The source code is available in https://github.com/Artanic30/HOICLIP.