Reviewerly/exHarmony huggingface dataset:
Multimodal Remote Sensing Image Matching Combining Learning Features and Delaunay Triangulation

Multimodal remote sensing image (MRSI) high-precision matching faces significant challenges due to the large nonlinear distortions in radiation and geometry. To address this problem, we propose a novel MRSI matching method, multimodal remote sensing image matching with Delaunay triangulation constraint network (M2DT-Net), which takes advantage of deep convolutional neural networks (CNNs) and the Delaunay triangulation (DT) strategy. First, a fine-tuned CNN model is applied to learn invariant features with high robustness across MRSIs. Second, an adaptive feature descriptor distance constraint combined with the DEGENSAC refinement is employed to filter the obtained matches, and only the matches with high credibility are retained. Finally, a DT strategy is adopted to expand more underlying correct matches that are dropped inliers in the previous stage. The experimental results on seven types of MRSI pairs indicate that M2DT-Net is superior to seven state-of-the-art image matching algorithms in terms of model performance and efficiency, achieving a balance of robustness and time consumption. The image registration results further demonstrate the effectiveness of the proposed algorithm. Therefore, the proposed M2DT-Net method provides a reference for resolving common multimodal image matching problems.
Uncertainty-Aware Adaptation for Self-Supervised 3D Human Pose Estimation

The advances in monocular 3D human pose estimation are dominated by supervised techniques that require large-scale 2D/3D pose annotations. Such methods often behave erratically in the absence of any provision to discard unfamiliar out-of-distribution data. To this end, we cast the 3D human pose learning as an unsupervised domain adaptation problem. We introduce MRP-Net <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Project page: https://sites.google.com/view/mrp-net that constitutes a common deep network backbone with two output heads subscribing to two diverse configurations; a) model-free Joint localization and b) model-based parametric regression. Such a design allows us to derive suitable measures to quantify prediction uncertainty at both pose and Joint level granularity. While supervising only on labeled synthetic samples, the adaptation process aims to minimize the uncertainty for the unlabeled target images while maximizing the same for an extreme out-of-distribution dataset (backgrounds). Alongside synthetic-to-real 3D pose adaptation, the Joint-uncertainties allow expanding the adaptation to work on in-the-wild images even in the presence of occlusion and truncation scenarios. We present a comprehensive evaluation of the proposed approach and demonstrate state-of-the-art performance on benchmark datasets.
Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation

Referring video object segmentation aims to predict foreground labels for objects referred by natural language expressions in videos. Previous methods either depend on 3D ConvNets or incorporate additional 2D ConvNets as encoders to extract mixed spatial-temporal features. However, these methods suffer from spatial misalignment or false distractors due to delayed and implicit spatial-temporal interaction occurring in the decoding phase. To tackle these limitations, we propose a Language-Bridged Duplex Transfer (LBDT) module which utilizes language as an intermediary bridge to accomplish explicit and adaptive spatial-temporal interaction earlier in the encoding phase. Concretely, cross-modal attention is performed among the temporal encoder, referring words and the spatial encoder to aggregate and transfer language-relevant motion and appearance information. In addition, we also propose a Bilateral Channel Activation (BCA) module in the decoding phase for further denoising and highlighting the spatial-temporal consistent features via channel-wise activation. Extensive experiments show our method achieves new state-of-the-art performances on four popular benchmarks with 6.8% and 6.9% absolute AP gains on A2D Sentences and J-HMDB Sentences respectively, while consuming around 7× less computational overhead <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/dzh19990407/LBDT.
Trajectory Optimization for Physics-Based Reconstruction of 3d Human Pose from Monocular Video

We focus on the task of estimating a physically plausi-ble articulated human motion from monocular video. Ex-isting approaches that do not consider physics often pro-duce temporally inconsistent output with motion artifacts, while state-of-the-art physics-based approaches have either been shown to work only in controlled laboratory conditions or consider simplified body-ground contact limited to feet. This paper explores how these shortcomings can be addressed by directly incorporating a fully-featured physics engine into the pose estimation process. Given an uncon-trolled, real-world scene as input, our approach estimates the ground-plane location and the dimensions of the physi-cal body model. It then recovers the physical motion by per-forming trajectory optimization. The advantage of our for-mulation is that it readily generalizes to a variety of scenes that might have diverse ground properties and supports any form of self-contact and contact between the articu-lated body and scene geometry. We show that our approach achieves competitive results with respect to existing physics-based methods on the Human3.6M benchmark [13], while being directly applicable without re-training to more complex dynamic motions from the AIST benchmark [36] and to uncontrolled internet videos.
Lightweight Privacy-Preserving Spatial Keyword Query over Encrypted Cloud Data

With the rapid development of geographic location technology and the explosive growth of data, a large amount of spatio-textual data is outsourced to the cloud server to reduce the local high storage and computing burdens, but at the same time causes security issues such as data privacy leakage. Thus, extensive privacy-preserving spatial keyword query schemes have been proposed. Most of the existing schemes use Asymmetric Scalar-Product-Preserving Encryption (ASPE) for encryption, but ASPE has proven to be insecure. And the existing spatial range query schemes require users to provide more information about the query range and generate a large amount of ciphertext, which causes high storage and computational burdens. To solve these issues, in this paper we introduce some random numbers and a random permutation to enhance the security of ASPE scheme, and then propose a novel privacy-preserving Spatial Keyword Query (SKQ) scheme based on the enhanced ASPE and Geohash algorithm. In addition, we design a more Lightweight Spatial Keyword Query (LSKQ) scheme by using a unified index for spatial range and multiple keywords, which not only greatly decreases SKQ’s storage and computational costs but also requires users to provide little information about query region. Finally, formal security analysis proves that our schemes have Indistinguishability under Chosen Plaintext Attack (IND-CPA), and extensive experiments demonstrate that our enhanced scheme is efficient and practical.
Physically-Based Editing of Indoor Scene Lighting from a Single Image

We present a method to edit complex indoor lighting from a single image with its predicted depth and light source segmentation masks. This is an extremely challenging problem that requires modeling complex light transport, and disentangling HDR lighting from material and geometry with only a partial LDR observation of the scene. We tackle this problem using two novel components: 1) a holistic scene reconstruction method that estimates reflectance and parametric 3D lighting, and 2) a neural rendering framework that re-renders the scene from our predictions. We use physically-based light representations that allow for intuitive editing, and infer both visible and invisible light sources. Our neural rendering framework combines physically-based direct illumination and shadow rendering with deep networks to approximate global illumination. It can capture challenging lighting effects, such as soft shadows, directional lighting, specular materials, and interreflections. Previous single image inverse rendering methods usually entangle lighting and geometry and only support applications like object insertion. Instead, by combining parametric 3D lighting estimation with neural scene rendering, we demonstrate the first automatic method for full scene relighting from a single image, including light source insertion, removal, and replacement.
Surveying the developer experience of flaky tests

Test cases that pass and fail without changes to the code under test are known as flaky. The past decade has seen increasing research interest in flaky tests, though little attention has been afforded to the views and experiences of software developers. In this study, we utilized a multi-source approach to obtain insights into how developers define flaky tests, their experiences of the impacts and causes of flaky tests, and the actions they take in response to them. To that end, we designed a literature-guided developer survey that we deployed on social media, receiving 170 total responses. We also searched on StackOverflow and analyzed 38 threads relevant to flaky tests, offering a distinct perspective free of any self-reporting bias. Through a mixture of numerical and thematic analyses, this study reveals a number of findings, including (1) developers strongly agree that flaky tests hinder continuous integration; (2) developers who experience flaky tests more often may be more likely to ignore potentially genuine test failures; and (3) developers rate issues in setup and teardown to be the most common causes of flaky tests.
On Adapting the DIET Architecture and the Rasa Conversational Toolkit for the Sentiment Analysis Task

The Rasa open-source toolkit provides a valuable Natural Language Understanding (NLU) infrastructure to assist the development of conversational agents.In this paper, we show that this infrastructure can seamlessly and effectively be used for other different NLU-related text classification tasks, such as sentiment analysis.The approach is evaluated on three widely used datasets containing movie reviews, namely IMDb, Movie Review (MR) and the Stanford Sentiment Treebank (SST2).The results are consistent across the three databases, and show that even simple configurations of the NLU pipeline lead to accuracy rates that are comparable to those obtained with other state-of-the-art architectures.The best results were obtained when the Dual Intent and Entity Transformer (DIET) architecture was fed with pre-trained word embeddings, surpassing other recent proposals in the sentiment analysis field.In particular, accuracy rates of 0.907, 0.816 and 0.858 were obtained for the IMDb, MR and SST2 datasets, respectively.
Generalizable Human Pose Triangulation

We address the problem of generalizability for multi-view 3D human pose estimation. The standard approach is to first detect 2D keypoints in images and then apply triangulation from multiple views. Even though the existing methods achieve remarkably accurate 3D pose estimation on public benchmarks, most of them are limited to a single spatial camera arrangement and their number. Several methods address this limitation but demonstrate significantly degraded performance on novel views. We propose a stochastic framework for human pose triangulation and demonstrate a superior generalization across different camera arrangements on two public datasets. In addition, we apply the same approach to the fundamental matrix estimation problem, showing that the proposed method can successfully apply to other computer vision problems. The stochastic framework achieves more than 8.8% improvement on the 3D pose estimation task, compared to the state-of-the-art, and more than 30% improvement for fundamental matrix estimation, compared to a standard algorithm.
Defensive Patches for Robust Recognition in the Physical World

To operate in real-world high-stakes environments, deep learning systems have to endure noises that have been con-tinuously thwarting their robustness. Data-end defense, which improves robustness by operations on input data in-stead of modifying models, has attracted intensive attention due to its feasibility in practice. However, previous data-end defenses show low generalization against diverse noises and weak transferability across multiple models. Motivated by the fact that robust recognition depends on both local and global features, we propose a defensive patch generation framework to address these problems by helping mod-els better exploit these features. For the generalization against diverse noises, we inject class-specific identifiable patterns into a confined local patch prior, so that defensive patches could preserve more recognizable features towards specific classes, leading models for better recognition under noises. For the transferability across multiple models, we guide the defensive patches to capture more global fea-ture correlations within a class, so that they could activate model-shared global perceptions and transfer better among models. Our defensive patches show great potentials to im-prove application robustness in practice by simply sticking them around target objects. Extensive experiments show that we outperform others by large margins (improve 20+ % accuracy for both adversarial and corruption robustness on average in the digital and physical world). <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Our codes are available at https://github.com/nlsde-safety-team/DefensivePatch.
SCENIC: A JAX Library for Computer Vision Research and Beyond

Scenic is an open-source <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/google-research/scenic JAX library with a focus on transformer-based models for computer vision research and beyond. The goal of this toolkit is to facilitate rapid experimentation, prototyping, and research of new architectures and models. Scenic supports a diverse range of tasks (e.g., classification, segmentation, detection) and facilitates working on multi-modal problems, along with GPU/TPU support for large-scale, multi-host and multi-device training. Scenic also offers optimized implementations of state-of-the-art research models spanning a wide range of modalities. Scenic has been successfully used for numerous projects and published papers and continues serving as the library of choice for rapid prototyping and publication of new research ideas.
Hybrid-DANet: An Encoder-Decoder Based Hybrid Weights Alignment With Multi-Dilated Attention Network for Automatic Brain Tumor Segmentation

Gliomas are the most common and highly growing tumors lead to high mortality rate in their highest grade. The early diagnosis of gliomas, and treatment planning are most important steps to enhance the life expectancy of a patient. Among the modern imaging techniques, magnetic resonance imaging (MRI) is the most robust and widely used technique to visualize the brain tumor. The CNN-based networks mainly depend on multi-branch and increasing the depth/width of the network to enhance the segmentation accuracy at the cost of high computational cost. To mitigate these drawbacks we therefore, propose a hybrid weights alignment with multi-dilated attention network for automatic brain tumor segmentation (Hybrid-DANet). It employs multiple modules incorporated on baseline encoder-decoder architecture. Firstly, we proposed a novel hybrid weight alignment with multi-dilated attention module (HWADA) is used between the skip connections. It has capability to obtain the different sets of aligned weight by using different dilation schemes. Different weight alignments play a vital role to obtain very precise targeted information while negating the less informative part. It utilizes the low and high level information with skip connections across each branch of encoder and decoder. Secondly, we incorporated a multi channel multi scale module (MCS) on the baseline module. It consists of multiple channels used to extract the channel-wise information with more reduced computational cost. To reduce the resultant saturated accuracy due to vanishing gradient problem, we incorporated the residual module (RM). Thus, the RM, and MCS are useful to obtain the deep, intrinsic, channel-wise feature without expansion of depth and height. Whereas the novel HWADA not only propagates the low level information but also process it to obtain more semantic features used for decoder. We have tested our proposed technique on well-known datasets; BraTS 2017, and 2018 with comparable performance to counter-part.
STCrowd: A Multimodal Dataset for Pedestrian Perception in Crowded Scenes

Accurately detecting and tracking pedestrians in 3D space is challenging due to large variations in rotations, poses and scales. The situation becomes even worse for dense crowds with severe occlusions. However, existing benchmarks either only provide 2D annotations, or have limited 3D annotations with low-density pedestrian distribution, making it difficult to build a reliable pedestrian perception system especially in crowded scenes. To better evaluate pedestrian perception algorithms in crowded scenarios, we introduce a large-scale multimodal dataset, STCrowd. Specifically, in STCrowd, there are a total of 219 K pedestrian instances and 20 persons per frame on average, with various levels of occlusion. We provide synchronized LiDAR point clouds and camera images as well as their corresponding 3D labels and joint IDs. STCrowd can be used for various tasks, including LiDAR-only, image-only, and sensor-fusion based pedestrian detection and tracking. We provide baselines for most of the tasks. In addition, considering the property of sparse global distribution and density-varying local distribution of pedestrians, we further propose a novel method, Density-aware Hierarchical heatmap Aggregation (DHA), to enhance pedestrian perception in crowded scenes. Extensive experiments show that our new method achieves state-of-the-art performance for pedestrian detection on various datasets. https://github.com/4DVLab/STCrowd.git.
A Simple Data Mixing Prior for Improving Self-Supervised Learning

Data mixing (e.g., Mixup, Cutmix, ResizeMix) is an essential component for advancing recognition models. In this paper, we focus on studying its effectiveness in the self-supervised setting. By noticing the mixed images that share the same source images are intrinsically related to each other, we hereby propose SDMP, short for Simple Data Mixing Prior, to capture this straightforward yet essential prior, and position such mixed images as additional positive pairs to facilitate self-supervised representation learning. Our experiments verify that the proposed SDMP enables data mixing to help a set of self-supervised learning frameworks (e.g., MoCo) achieve better accuracy and out-of-distribution robustness. More notably, our SDMP is the first method that successfully leverages data mixing to improve (rather than hurt) the performance of Vision Transformers in the self-supervised setting. Code is publicly available at https://github.com/OliverRensu/SDMP.
Optimizing Video Prediction via Video Frame Interpolation

Video prediction is an extrapolation task that predicts future frames given past frames, and video frame interpolation is an interpolation task that estimates intermediate frames between two frames. We have witnessed the tremendous advancement of video frame interpolation, but the general video prediction in the wild is still an open question. Inspired by the photo-realistic results of video frame interpolation, we present a new optimization framework for video prediction via video frame interpolation, in which we solve an extrapolation problem based on an interpolation model. Our video prediction framework is based on optimization with a pretrained differentiable video frame interpolation module without the need for a training dataset, and thus there is no domain gap issue between training and test data. Also, our approach does not need any additional information such as semantic or instance maps, which makes our framework applicable to any video. Extensive experiments on the Cityscapes, KITTI, DAVIS, Middlebury, and Vimeo90K datasets show that our video prediction results are robust in general scenarios, and our approach outperforms other video prediction methods that require a large amount of training data or extra semantic information.
Joint Video Summarization and Moment Localization by Cross-Task Sample Transfer

Video summarization has recently engaged increasing attention in computer vision communities. However, the scarcity of annotated data has been a key obstacle in this task. To address it, this work explores a new solution for video summarization by transferring samples from a correlated task (i.e., video moment localization) equipped with abundant training data. Our main insight is that the annotated video moments also indicate the semantic highlights of a video, essentially similar to video summary. Approximately, the video summary can be treated as a sparse, redundancy-free version of the video moments. Inspired by this observation, we propose an importance Propagation based collaborative Teaching Network (iPTNet). It consists of two separate modules that conduct video summarization and moment localization, respectively. Each module estimates a frame-wise importance map for indicating keyframes or moments. To perform cross-task sample transfer, we devise an importance propagation module that realizes the conversion between summarization-guided and localization-guided importance maps. This way critically enables optimizing one of the tasks using the data from the other task. Additionally, in order to avoid error amplification caused by batch-wise joint training, we devise a collaborative teaching scheme, which adopts a crosstask mean teaching strategy to realize the joint optimization of the two tasks and provide robust frame-level teaching signals. Extensive experiments on video summarization benchmarks demonstrate that iPTNet significantly outperforms previous state-of-the-art video summarization methods, serving as an effective solution that overcomes the data scarcity issue in video summarization.