Reviewerly/exHarmony huggingface dataset:
Cloud-Based Parallel Tiling Algorithm for Large Scale Remote Sensing Datasets

Tiled remote sensing data is essential for web-based remote sensing applications, widely applied in online map services, cloud-based remote sensing processing, etc. However, most existing tiling algorithms focus on tiling single remote sensing images with stand-alone machines. As the volume of remote sensing data increases, the demand for tiling high-resolution and large-scale remote sensing datasets increases dramatically. In this research, we propose a cloud-based parallel tiling algorithm for large-scale remote sensing datasets. A three-step processing flow is designed to implement the tiling of datasets composed of a set of images. Furthermore, three types of cloud-based storage are adopted to improve the efficiency of data extraction, namely, cloud storage, block storage, and NoSQL. We experimented with the proposed algorithm for tiling a national-scale 2 m resolution remote sensing dataset. The whole process took about 25.7 hours with up to 180 cores.
Learning Multi-Context Dynamic Listwise Relation for Generalizable Person Re-Identification

Although Person re-identification (Re-ID) has made rapid development in supervised learning and domain adaptation, it is more desirable to learn a generalizable model that can be directly applied to unseen scenes without updating. Generalizable Re-ID is challenging due to uncertain cross-camera variations in unseen target domain, such as illumination and viewpoint change, which result in visual ambiguities. Existing generalizable Re-ID methods focus on learning more generalizable features for individual instances. They ignore the context in the ranking list of the target domain. When human encounters visual ambiguities when matching pedestrians in unfamiliar scenes, comparing similar instances in the ranking list and comparing environments in different cameras can help remove ambiguities and refine matching results. This is actually exploiting contextual information in target domain, which is ignored by existing generalizable Re-ID methods. For learning contextual information to refine matching in unseen target domain, we propose a Multi-Context Dynamic Listwise Relation Network (MDLRN) to extract and aggregate instance-level and camera-level contextual features of a list of images, which can dynamically adapt the metric to unseen cross-domain scene variations. We further propose camera-specific feature perturbation (CFP) to simulate cross-camera variations in unseen target domain to improve generalization. Extensive experiments showed the superiority of our method in domain generalization.
Python: Data Handling, Analysis and Plotting

Abstract When performing an image analysis pipeline, a programming language like Python is mainly used for two distinctive applications: (1) the analysis of the acquired images, such as background removal, noise reduction, object segmentation, measurements of biological structures and events, etc. and (2) the analysis of the data obtained as a result of the image analysis, such as a calculating a histogram from the noise-removed image or statistics on the shape of the segmented object.
Visual Question Answering of Remote Sensing Image Based on Attention Mechanism

In recent years, the research of attention mechanism has made significant progress in the field of computer vision. In the processing of visual problems of remote sensing images, the attention mechanism can make the computer focus on important image areas and improve the accuracy of question answering. Our research focuses on the role of synergistic attention mechanisms in the interaction of question representations and visual representations. On the basis of Modular Collaborative Attention (MCA), according to the complementary characteristics of global features and local features, the hybrid connection strategy is used to perceive global features at the same time without weakening the attention distribution of local features. The impact of attention mechanisms on various types of visual question answering questions has been evaluated:(i) scene classification (ii)object comparison (iii) quantitative statistics (iv) relational judgment. By fusing the global features and local features of different modalities, the model can obtain more information between modalities. Model performance evaluation under the RSVQA-LR dataset. Experimental results show, the method in this paper improves the global accuracy by 9.81% than RSVQA.
Hybrid Microarray Gene Data Classification Based on GA-ACO Optimization

The advancement in the computerized medical scenario has grabbed the attention of researchers working in the field of medicine. Since clinical decision-making demands the utmost accuracy of diagnosis, it is a tedious and challenging task for physicians. An automated system that helps in disease diagnosis will benefit the medical industry. Microarray gene data classification is currently the most common cancer among world- wide. In this research work, a hybrid classification of genetic algorithm with Ant Colony Optimization (HGAACO) model for knowledge mining from microarray gene dataset is proposed and evaluated. The proposed HGAACO Classifier is one of the data mining techniques used to make decisions on real gene data. Using data directly from the database may affect the performance of the system. Pre-processing provides the data in a desirable form by handling missing values, selecting features, and scaling of datasets. Classification of the processed data provides a better decision on diagnosis. ACO optimizes and evaporation gradually reduces the pheromone level of all the trails. Subsequently, the pheromone levels of the trails that are not followed gradually decrease, which in turn lowers the probability of the trail being chosen by subsequent ants. HGAACO accuracy of 94.6% and a maximum accuracy improvement of 3.0% of other conventional approaches.
A Multi-station Signal Sorting Method Based on Time Difference Pair and Azimuth Information

Current sorting methods based on time difference of arrival (TDOA) can not concurrently realize the high sorting accuracy and low computational complexity with pulse loss and interference. This paper proposes a multi-station signal sorting method based on time difference pair and azimuth information. Firstly, the time difference window is introduced to determine the pulse range in which the pulse may be correctly matched with the pulse in the selected master station. According to the multi-parameters information, pulses in the range are accurately matched to get the real pulse pair and the time difference pair. Finally, the received signals are sorted by the time difference pair and azimuth information within the error tolerance. The simulation results show that the proposed method can effectively sort the interleaved signals with the pulse loss, interference, and the received pulses with few numbers. Furthermore, the computational complexity of the proposed method is greatly reduced.
Generative-Network Based Multimedia Super-Resolution for Uav Remote Sensing

Unmanned Aerial Vehicle (UAV) based aerial mapping has taken over the surveying industry thanks to low costs and ease of use. Although these UAVs have relatively high-resolution imaging systems, there exists a near exponential relationship between the ground sampling distance (GSD) and the number of images required - which is a function of flight altitude. To tackle this, we use a generative network based super-resolution approach to increase the GSD of images which effectively reduces flight time. In this paper we test the efficiency and efficacy of this approach using two multimedia super-resolution implementations. We also provide quantitative results comparing the two using various image processing metrics.
Driving in the Virtual: Towards Large-Scale Software-Defined Transportation Scene

Image dataset plays an essential role in training the computer vision models. Compared with realistic datasets that are expensive to label and inflexible in number and scene, virtual datasets gradually show advantages and begin to act as a supplement in the training and validation of computer vision models. In this paper, we propose a pipeline for constructing artificial traffic scenes and generating virtual datasets based on an automatic driving simulation software from the perspective of parallel vision. Firstly, map data and simulation modeling elements like buildings are used to model the 3D artificial scene built upon a real environment. Furthermore, we use those artificial transportation scenes to automatically generate a large-scale and diverse dataset with the ground-truth label, which can provide service for computer vision tasks such as object detection and semantic segmentation. The environmental conditions and sensor properties in artificial scenes can be controlled flexibly and repeatedly. Finally, we show the detail and performance of creating virtual artificial scenes and generating the virtual dataset with low modeling time and highly accurate labeling, which does not depend on manual labeling and achieves a diverse dataset automatically and conveniently.
Semantic Attention Network for Image Captioning and Visual Question Answering Based on Image High-Level Semantic Attributes

The main challenge in the vision-to-language system is generation of the caption with a proper meaningful answer for a question and extracting even the minute details from the image. The main contributions in this paper are presenting an approach based on image high-level semantic attributes and local image features address the challenges of V2L tasks. Especially, the high-level semantic attributes information is used to reduce the semantic gap between images and text. A novel semantic attention network is designed to explore the mapping relationships between semantic attributes and image regions. The semantic attention network highlights the concept-related regions and selects the region-related concepts. Two special V2L tasks, image captioning and VQA, are addressed by the proposed approach. Improved BLEU score shows the proposed image captioning performs well. The experimental results show that the proposed model is effective for V2L tasks.
Edge-enhanced efficient network for remote sensing image super-resolution

The super-resolution (SR) reconstruction is drawing increasing attention in remote-sensing image processing, owing to improving the spatial resolution and enriching the details of initially obtaining low-resolution (LR) images. Different from natural images, remote sensing images usually have more complex image distribution and degradation processes. Although current deep convolutional neural network (DCNN)-based approaches reach better performance by deepening the network and introducing attention mechanism, this is usually accompanied by more parameters, more computation, and more complex designs. To improve the performance and efficiency of SR reconstruction of remote sensing images, we proposed the edge-enhanced efficient network (EESR). Specifically, we design an inception-like multi-branch convolution block, named edge-enhanced convolution block (EEB), to enrich edge-aware capabilities of the network, which includes multi-order gradient extraction and other feature enhancement branches. Furthermore, re-parameterization is introduced into the inference stage of the network, aiming at boosting efficient inference. We re-parameterize the weights of EEB to the edge-enhanced convolution (EEC) via equivalent transformation to reduce the parameters and computational cost of the inference model. In addition, we construct a paired SR dataset named GF2-Port for remote sensing images with degradation simulation, based on the data of GaoFen-2 port and strait. Extensive experiments on both established and public datasets indicate that the proposed EESR outperforms other comparable approaches in terms of balancing restoration accuracy and inference efficiency.
Temporal Multi-Agents’ Logics Modelling Dynamic Temporal Accessibility Relations

Here we continue our research connected with multi-agent logics and theirs applications to Information Sciences. We work with multi-agent logics based at relational models where time accessibility relations are not compulsory transitive and not compulsory are solid or continuous. We admit that time accessibility relations may have gaps or places of forgotten time. Several various approaches to formalizations paths of accessibility by time are suggested. Among them the case when each time state generates its own time - an interval of time which states are accepted as all the time accessible for any given agent in that time state. Variation of this approach when an external modelling initially introduced as all the times forgotten for the agent and not accessible for this agent at all. Also we study the case with time sates imitated as clusters of information states for current time. Examples of representation important properties concerning knowledge and information via logical formulas are provided. Mathematical results are formulated in terms of decidability and satisfiability problem, and existence of corresponding algorithms are stated. Open problems are suggested in the conclusion.
Defect Detection of Fresco Samples using Digital Holography

A novel portable digital holographic speckle pattern interferometer instrument was developed to detect scathes of fresco samples. Results show that the proposed technique achieves a high accuracy of 3D imaging of the sample.
3D Human Pose and Shape Estimation from Video

Despite the state-of-the-art video-based 3D human pose and shape estimation methods have achieved great progress in improving the temporal consistency of prediction results, there is a certain gap in the accuracy of 3D pose prediction on in- the-wild dataset compared with the state-of-the-art single image- based methods. The first reason is that these single image-based methods can benefit from in-the-wild single image datasets with high-precision 3D pseudo annotations, and the other one is the limitation of iterative error feedback (IEF) loop structure itself, which is a parameter regressor. In this paper, we propose a new neural network framework for 3D human pose and shape estimation from video. We first design two recurrent encoders based on Convolutional Gated Recurrent Unit Network (ConvGRU) to extract volumetric features with temporal information, and then a part attention mechanism is used to get final features, finally we use a regressor to obtain the Skinned Multi-Person Linear (SMPL) model parameters. We also design our method to support the hybrid training on video datasets and single image datasets, so that our method can benefit from these in-the-wild single image datasets. Experimental results show that our method outperforms state-of-the-art video-based methods on in-the-wild dataset (3DPW) without any fine-tuning.
Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning

Deep reinforcement learning (DRL) is a promising approach to solve complex control tasks by learning policies through interactions with the environment. However, the training of DRL policies requires large amounts of training experiences, making it impractical to learn the policy directly on physical systems. Sim-to-real approaches leverage simulations to pretrain DRL policies and then deploy them in the real world. Unfortunately, the direct real-world deployment of pretrained policies usually suffers from performance deterioration due to the different dynamics, known as the reality gap. Recent sim-to-real methods, such as domain randomization and domain adaptation, focus on improving the robustness of the pretrained agents. Nevertheless, the simulation-trained policies often need to be tuned with real-world data to reach optimal performance, which is challenging due to the high cost of real-world samples. This work proposes a distributed cloud-edge architecture to train DRL agents in the real world in real-time. In the architecture, the inference and training are assigned to the edge and cloud, separating the real-time control loop from the computationally expensive training loop. To overcome the reality gap, our architecture exploits sim-to-real transfer strategies to continue the training of simulation-pretrained agents on a physical system. We demonstrate its applicability on a physical inverted-pendulum control system, analyzing critical parameters. The real-world experiments show that our architecture can adapt the pretrained DRL agents to unseen dynamics consistently and efficiently.
A Joint Model Based on Prior Knowledge for War Research Report Information Extraction

War Research Reports (WRR) objectively describes the composition, deployment and cooperation of troops and weapons in the war, as well as the combat damage. WRR information extraction can provide data for the knowledge graph of damage. The War Research Report Dataset (WRRD) is constructed based on manual tagging and WRR collected on the Internet, then a joint entity and relation extraction model named PRK-CASREL based on prior knowledge for WRR is proposed. The result of the experiment indicates that shows that the proposed model has better recall and F1 than the models involved in comparison.
Efficient Shadow Removal and Enhancement of Light Texts in Scanned Documents

The advancement of high-resolution cameras in mobile devices enables users to capture high quality document images using such devices. The environmental lighting conditions and different illuminations introduce shadows in the captured document images. In this paper, we present a novel algorithm for shadow removal based on a shadow mask computed using a deep learning model. The existing shadow removal process degrades the quality of text in documents, especially the light texts. To avoid text degradation, the shadow removal process has been controlled on the text regions using a document foreground mask of the input image computed using an adaptive thresholding technique. The texts are further enhanced using non-linear contrast stretching technique. The proposed shadow removal and text enhancement algorithms have been compared with the existing algorithms in the literature and the experimental results shows the efficacy of the algorithms. The accuracy and low computation complexity of the algorithms make it suitable to use in mobile applications.
Domain Randomization with Adaptive Weight Distillation

Domain Randomization (DR) is one of the increasingly popular techniques for domain generalization, which learns a policy from simulation by randomizing domain parameters. However, integrating information from randomized domains into one policy may lead to high variance and an unstable training process. Thus, we draw on policy distillation, distilling multiple policies into a single policy to effectively reduce the high variance of the model. The following question is whether the importance of multiple tasks may affect the performance of the distilled policy. To address this issue, we propose an adaptive-weight distillation strategy depending on teacher policy performance called Domain Randomization with Adaptive Weight Distillation (DRAWD) to control the student policy to learn toward better-performing teacher policy. This way, DRAWD addresses the problem of the under-generalization of a single policy and the potential imperfect teacher of a multiple-policy ensemble. We compare DRAWD with two baselines on two Mujoco continuous control tasks. Our results show that the target domain performance of policies trained with DRAWD is better than the other two baselines on the metric of variance and average rewards of the task.
Scalable Learning to Optimize: A Learned Optimizer Can Train Big Models

Learning to optimize (L2O) has gained increasing attention since it demonstrates a promising path to automating and accelerating the optimization of complicated problems. Unlike manually crafted classical optimizers, L2O parameterizes and learns optimization rules in a data-driven fashion. However, the primary barrier, scalability, persists for this paradigm: as the typical L2O models create massive memory overhead due to unrolled computational graphs, it disables L2O's applicability to large-scale tasks. To overcome this core challenge, we propose a new scalable learning to optimize (SL2O) framework which (i) first constrains the network updates in a tiny subspace and (ii) then explores learning rules on top of it. Thanks to substantially reduced trainable parameters, learning optimizers for large-scale networks with a single GPU become feasible for the first time, showing that the scalability roadblock of applying L2O to training large models is now removed. Comprehensive experiments on various network architectures (i.e., ResNets, VGGs, ViTs) and datasets (i.e., CIFAR, ImageNet, E2E) across vision and language tasks, consistently validate that SL2O can achieve significantly faster convergence speed and competitive performance compared to analytical optimizers. For example, our approach converges 3.41 $$\sim $$ 4.60 times faster on CIFAR-10/100 with ResNet-18, and 1.24 times faster on ViTs, at nearly no performance loss. Codes are in https://github.com/VITA-Group/Scalable-L2O .
Task Decoupling in Preference-based Reinforcement Learning for Personalized Human-Robot Interaction

Intelligent robots designed to interact with hu-mans in the real world need to adapt to the preferences of different individuals. Preference-based reinforcement learning (RL) has shown great potential for teaching robots to learn personalized behaviors from interacting with humans with-out a meticulous, hand-crafted reward function, replaced by learning reward based on a human's preferences between two robot trajectories. However, poor feedback efficiency and poor exploration in the state and reward spaces make current preference-based RL algorithms perform poorly in complex interactive tasks. To improve the performance of preference-based RL, we incorporate prior knowledge of the task into preference-based RL. Specifically, we decouple the task from preference in human-robot interaction. We utilize a sketchy task reward derived from task priori to instruct robots to conduct more effective task exploration. Then a learned reward from preference-based RL is used to optimize the robot's policy to align with human preferences. In addition, these two parts are combined organically via reward shaping. The experimental results show that our method is a practical and effective solution for personalized human-robot interaction. Code is available at https://github.com/Wenminggong/PbRL_for_PHRI.
Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input

The ability to model intra-modal and inter-modal interactions is fundamental in multimodal machine learning. The current state-of-the-art models usually adopt deep learning models with fixed structures. They can achieve exceptional performances on specific tasks, but face a particularly challenging problem of modality mismatch because of diversity of input modalities and their fixed structures. In this paper, we present Switch-BERT for joint vision and language representation learning to address this problem. Switch-BERT extends BERT architecture by introducing learnable layer-wise and cross-layer interactions. It learns to optimize attention from a set of attention modes representing these interactions. One specific property of the model is that it learns to attend outputs from various depths, therefore mitigates the modality mismatch problem. We present extensive experiments on visual question answering, image-text retrieval and referring expression comprehension experiments. Results confirm that, whereas alternative architectures including ViLBERT and UNITER may excel in particular tasks, Switch-BERT can consistently achieve better or comparable performances than the current state-of-the-art models in these tasks. Ablation studies indicate that the proposed model achieves superior performances due to its ability in learning task-specific multimodal interactions.