Reviewerly/exHarmony huggingface dataset:
NFormer: Robust Person Re-identification with Neighbor Transformer

Person re-identification aims to retrieve persons in highly varying settings across different cameras and scenarios, in which robust and discriminative representation learning is crucial. Most research considers learning representations from single images, ignoring any potential interactions between them. However, due to the high intraidentity variations, ignoring such interactions typically leads to outlier features. To tackle this issue, we propose a Neighbor Transformer Network, or NFormer, which explicitly models interactions across all input images, thus suppressing outlier features and leading to more robust representations overall. As modelling interactions between enormous amount of images is a massive task with lots of distractors, NFormer introduces two novel modules, the Landmark Agent Attention, and the Reciprocal Neighbor Softmax. Specifically, the Landmark Agent Attention efficiently models the relation map between images by a low-rank factorization with a few landmarks in feature space. Moreover, the Reciprocal Neighbor Softmax achieves sparse attention to relevant -rather than all- neighbors only, which alleviates interference of irrelevant representations and further relieves the computational burden. In experiments on four large-scale datasets, NFormer achieves a new state-of-the-art. The code is released at https://github.com/haochenheheda/NFormer.
IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo

We present IterMVS, a new data-driven method for high-resolution multi-view stereo. We propose a novel GRU-based estimator that encodes pixel-wise probability distributions of depth in its hidden state. Ingesting multi-scale matching information, our model refines these distributions over multiple iterations and infers depth and confidence. To extract the depth maps, we combine traditional classification and regression in a novel manner. We verify the efficiency and effectiveness of our method on DTU, Tanks&Temples and ETH3D. While being the most efficient method in both memory and run-time, our model achieves competitive performance on DTU and better generalization ability on Tanks&Temples as well as ETH3D than most state-of-the-art methods. Code is available at https://github.com/FangjinhuaWang/IterMVS.
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation

Unsupervised domain adaptation (UDA) aims to adapt a model trained on the source domain (e.g. synthetic data) to the target domain (e.g. real-world data) without requiring further annotations on the target domain. This work focuses on UDA for semantic segmentation as real-world pixel-wise annotations are particularly expensive to acquire. As UDA methods for semantic segmentation are usually GPU memory intensive, most previous methods operate only on downscaled images. We question this design as low-resolution predictions often fail to preserve fine details. The alternative of training with random crops of high-resolution images alleviates this problem but falls short in capturing long-range, domain-robust context information. Therefore, we propose HRDA, a multi-resolution training approach for UDA, that combines the strengths of small high-resolution crops to preserve fine segmentation details and large low-resolution crops to capture long-range context dependencies with a learned scale attention, while maintaining a manageable GPU memory footprint. HRDA enables adapting small objects and preserving fine segmentation details. It significantly improves the state-of-the-art performance by 5.5 mIoU for GTA $$\rightarrow $$ Cityscapes and 4.9 mIoU for Synthia $$\rightarrow $$ Cityscapes, resulting in unprecedented 73.8 and 65.8 mIoU, respectively. The implementation is available at github.com/lhoyer/HRDA .
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound

As humans, we navigate a multimodal world, building a holistic understanding from all our senses. We introduce @MERLOT RESERVE, a model that represents videos jointly over time - through a new training objective that learns from audio, subtitles, and video frames. Given a video, we replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale: we pretrain on 20 million YouTube videos. Empirical results show that @MERLOT RESERVE learns strong multimodal representations. When finetuned, it sets state-of-the-art on Visual Commonsense Reasoning (VCR), TVQA, and Kinetics-600; outperforming prior work by 5%, 7%, and 1.5% respectively. Ablations show that these tasks benefit from audio pretraining - even VCR, a QA task centered around images (without sound). Moreover, our objective enables out-of-the-box prediction, revealing strong multimodal commonsense understanding. In a fully zero-shot setting, our model obtains competitive results on four video tasks, even outperforming supervised approaches on the recently proposed Situated Reasoning (STAR) benchmark. We analyze why audio enables better vision-language representations, suggesting significant opportunities for future research. We conclude by discussing ethical and societal implications of multimodal pretraining.
SimVP: Simpler yet Better Video Prediction

From CNN, RNN, to ViT, we have witnessed remarkable advancements in video prediction, incorporating auxiliary inputs, elaborate neural architectures, and sophisticated training strategies. We admire these progresses but are confused about the necessity: is there a simple method that can perform comparably well? This paper proposes SimVp, a simple video prediction model that is completely built upon CNN and trained by MSE loss in an end-to-end fashion. Without introducing any additional tricks and complicated strategies, we can achieve state-of-the-art performance on five benchmark datasets. Through extended experiments, we demonstrate that SimVP has strong generalization and extensibility on real-world datasets. The significant reduction of training cost makes it easier to scale to complex scenarios. We believe SimVP can serve as a solid baseline to stimulate the further development of video prediction.
Metaverse and Sustainability: Systematic Review of Scientific Publications until 2022 and Beyond

The metaverse is the next evolution of the Internet, merging the physical and digital worlds into a multiuser environment. It is discussed widely in the media and among technology developers. It may expand to many aspects of society and people’s everyday lives. In this article, we examine how academic discussion and research about the metaverse developed from the 1990s to the end of 2022. We focus on the quantitative development of scientific publications about the metaverse, the key countries and organizations behind these publications, the key research topics and areas, and whether and how those publications addressed sustainability. We identified 491 international scientific publications (peer reviewed articles, reviews, and proceedings papers) related to the metaverse in the Web of Science database and 2240 scientific publications in the Scopus database, between 1995 and 2022. The number of publications is rising very fast as most of publications on the metaverse were published in 2022. Scholars in universities and research institutes in the United States, China, the United Kingdom, and South Korea are the most frequent publishers. Publications very seldomly address sustainability as the main subject. Usually, sustainability is considered very narrowly, despite the metaverse’s large and significant expected future economic and social impact. Sustainability and responsibility should be integrated into the design, construction, and development of the metaverse and related research.
Encryption and steganography-based text extraction in IoT using the EWCTS optimizer

This paper develops an effective encryption and steganography-based text extraction in IoT using deep learning method. Initially, the input text and cover images are separately pre-processed. DCT (discrete cosine transform) is utilized to transfer the image from spatial domain to frequency domain. Then, the original text is encrypted using new optimized equilibrium-based homomorphic encryption (OEHE) approach. Next, the extended wavelet convolutional transient search (EWCTS) optimizer with quotient multi-pixel value differencing (QMPVD) is developed to embed the secret text in cover images. Then, at receiver side, the reverse process for encryption and steganography is executed with secret key provided by the sender. Finally, the accurate text is extracted at receiver side using steganalysis process. The developed approach is executed in MATLAB software. The various evaluation metrics are used to authorize the effectiveness of suggested approach. Simulation outcomes proved that the suggested technique provides better outcomes than other existing approaches.
Dynamic Candidate Solution Boosted Beluga Whale Optimization Algorithm for Biomedical Classification

In many fields, complicated issues can now be solved with the help of Artificial Intelligence (AI) and Machine Learning (ML). One of the more modern Metaheuristic (MH) algorithms used to tackle numerous issues in various fields is the Beluga Whale Optimization (BWO) method. However, BWO has a lack of diversity, which could lead to being trapped in local optimaand premature convergence. This study presents two stages for enhancing the fundamental BWO algorithm. The initial stage of BWO’s Opposition-Based Learning (OBL), also known as OBWO, helps to expedite the search process and enhance the learning methodology to choose a better generation of candidate solutions for the fundamental BWO. The second step, referred to as OBWOD, combines the Dynamic Candidate Solution (DCS) and OBWO based on the k-Nearest Neighbor (kNN) classifier to boost variety and improve the consistency of the selected solution by giving potential candidates a chance to solve the given problem with a high fitness value. A comparison study with present optimization algorithms for single-objective bound-constraint optimization problems was conducted to evaluate the performance of the OBWOD algorithm on issues from the 2022 IEEE Congress on Evolutionary Computation (CEC’22) benchmark test suite with a range of dimension sizes. The results of the statistical significance test confirmed that the proposed algorithm is competitive with the optimization algorithms. In addition, the OBWOD algorithm surpassed the performance of seven other algorithms with an overall classification accuracy of 85.17% for classifying 10 medical datasets with different dimension sizes according to the performance evaluation matrix.
A Chaotic Image Encryption Method Based on the Artificial Fish Swarms Algorithm and the DNA Coding

Aiming at the problems of small key space and weak resistance to differential attacks in existing encryption algorithms, we proposed a chaotic digital image encryption scheme based on an optimized artificial fish swarm algorithm and DNA coding. First, the key is associated with the ordinary image pixel through the MD5 hash operation, and the hash value generated by the ordinary image is used as the initial value of the hyper-chaotic system to increase the sensitivity of the key. Next, the artificial fish school algorithm is used to scramble the positions of pixels in the block. In addition, scrambling operation between blocks is proposed to increase the scrambling effect. In the diffusion stage, operations are performed based on DNA encoding, obfuscation, and decoding technologies to obtain encrypted images. The research results show that the optimized artificial fish swarm algorithm has good convergence and can obtain the global optimal solution to the greatest extent. In addition, simulation experiments and security analysis show that compared with other encryption schemes, the scheme proposed in this paper has a larger key space and better resistance to differential attacks, indicating that the proposed algorithm has better encryption performance and higher security.
Everything is connected: Graph neural networks

In many ways, graphs are the main modality of data we receive from nature. This is due to the fact that most of the patterns we see, both in natural and artificial systems, are elegantly representable using the language of graph structures. Prominent examples include molecules (represented as graphs of atoms and bonds), social networks and transportation networks. This potential has already been seen by key scientific and industrial groups, with already-impacted application areas including traffic forecasting, drug discovery, social network analysis and recommender systems. Further, some of the most successful domains of application for machine learning in previous years-images, text and speech processing-can be seen as special cases of graph representation learning, and consequently there has been significant exchange of information between these areas. The main aim of this short survey is to enable the reader to assimilate the key concepts in the area, and position graph representation learning in a proper context with related fields.
Boosting R-CNN: Reweighting R-CNN samples by RPN’s error for underwater object detection

Complicated underwater environments bring new challenges to object detection, such as unbalanced light conditions, low contrast, occlusion, and mimicry of aquatic organisms. Under these circumstances, the objects captured by the underwater camera will become vague, and the generic detectors often fail on these vague objects. This work aims to solve the problem from two perspectives: uncertainty modeling and hard example mining. We propose a two-stage underwater detector named boosting R-CNN, which comprises three key components. First, a new region proposal network named RetinaRPN is proposed, which provides high-quality proposals and considers objectness and IoU prediction for uncertainty to model the object prior probability. Second, the probabilistic inference pipeline is introduced to combine the first-stage prior uncertainty and the second-stage classification score to model the final detection score. Finally, we propose a new hard example mining method named boosting reweighting. Specifically, when the region proposal network miscalculates the object prior probability for a sample, boosting reweighting will increase the classification loss of the sample in the R-CNN head during training, while reducing the loss of easy samples with accurately estimated priors. Thus, a robust detection head in the second stage can be obtained. During the inference stage, the R-CNN has the capability to rectify the error of the first stage to improve the performance. Comprehensive experiments on two underwater datasets and two generic object detection datasets demonstrate the effectiveness and robustness of our method. The link of code: https://github.com/mousecpn/Boosting-R-CNN.
EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications

In the pursuit of achieving ever-increasing accuracy, large and complex neural networks are usually developed. Such models demand high computational resources and therefore cannot be deployed on edge devices. It is of great interest to build resource-efficient general purpose networks due to their usefulness in several application areas. In this work, we strive to effectively combine the strengths of both CNN and Transformer models and propose a new efficient hybrid architecture EdgeNeXt. Specifically in EdgeNeXt, we introduce split depth-wise transpose attention (STDA) encoder that splits input tensors into multiple channel groups and utilizes depth-wise convolution along with self-attention across channel dimensions to implicitly increase the receptive field and encode multi-scale features. Our extensive experiments on classification, detection and segmentation tasks, reveal the merits of the proposed approach, outperforming state-of-the-art methods with comparatively lower compute requirements. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K, outperforming MobileViT with an absolute gain of 2.2% with 28% reduction in FLOPs. Further, our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K. The code and models are available at https://t.ly/_Vu9 .
Towards insighting cybersecurity for healthcare domains: A comprehensive review of recent practices and trends

Healthcare information security is becoming a significant responsibility for all healthcare organisations and individuals. Innovative medical equipment and healthcare apps are vital to patient care, yet they are often the target of hackers. Moreover, attackers are silently working against healthcare data. Once a hacker has gained access to a network, they might install ransomware to lock down essential services or encrypt files until a specified ransom is paid. Businesses are frequently compelled to pay the ransom, hoping the money is eventually recovered since the healthcare sector is time-sensitive. Although less common, network-connected devices can be taken over and used to distribute incorrect medications or alter a machine's functionality. So, there is a need to implement cyber security in healthcare to protect all information. In comparison to other industries, security duties in the healthcare industry are particularly broad and new. This is especially true given that data is accumulated and accessed from various destinations. Data on a specific patient is gathered from various sources, including hospital and lab records, insurance records, fitness apps, trackers and gadgets, health portals, and many more. It can be easily protected by using cybersecurity technology. This paper briefs about cybersecurity and its need in healthcare. Several tools, traits and roles of cybersecurity in the Healthcare Sector are studied. Finally, we identified and studied the applications of cybersecurity in healthcare. For hackers, a patient's aggregated data might be regarded as a goldmine, providing them with a detailed biography of an individual, including basic information, health trends, family history, and financial details. The importance of Data access in the healthcare sector emerges from numerous endpoints, which can be regarded as the weak spots of the healthcare data management system and can also open up an infringement in the medical data management infrastructure.
Evaluating the evidence for exponential quantum advantage in ground-state quantum chemistry

The idea to use quantum mechanical devices to simulate other quantum systems is commonly ascribed to Feynman. Since the original suggestion, concrete proposals have appeared for simulating molecular and materials chemistry through quantum computation, as a potential ``killer application''. Indications of potential exponential quantum advantage in artificial tasks have increased interest in this application, thus, it is critical to understand the basis for potential exponential quantum advantage in quantum chemistry. Here we gather the evidence for this case in the most common task in quantum chemistry, namely, ground-state energy estimation. We conclude that evidence for such an exponential advantage across chemical space has yet to be found. While quantum computers may still prove useful for quantum chemistry, it may be prudent to assume exponential speedups are not generically available for this problem.
A Modified YOLOv8 Detection Network for UAV Aerial Image Recognition

UAV multitarget detection plays a pivotal role in civil and military fields. Although deep learning methods provide a more effective solution to this task, changes in target size, shape change, occlusion, and lighting conditions from the perspective of drones still bring great challenges to research in this field. Based on the above problems, this paper proposes an aerial image detection model with excellent performance and strong robustness. First, in view of the common problem that small targets in aerial images are prone to misdetection and missed detection, the idea of Bi-PAN-FPN is introduced to improve the neck part in YOLOv8-s. By fully considering and reusing multiscale features, a more advanced and complete feature fusion process is achieved while maintaining the parameter cost as much as possible. Second, the GhostblockV2 structure is used in the backbone of the benchmark model to replace part of the C2f module, which suppresses information loss during long-distance feature transmission while significantly reducing the number of model parameters; finally, WiseIoU loss is used as bounding box regression loss, combined with a dynamic nonmonotonic focusing mechanism, and the quality of anchor boxes is evaluated by using “outlier” so that the detector takes into account different quality anchor boxes to improve the overall performance of the detection task. The algorithm’s performance is compared and evaluated on the VisDrone2019 dataset, which is widely used worldwide, and a detailed ablation experiment, contrast experiment, interpretability experiment, and self-built dataset experiment are designed to verify the effectiveness and feasibility of the proposed model. The results show that the proposed aerial image detection model has achieved obvious results and advantages in various experiments, which provides a new idea for the deployment of deep learning in the field of UAV multitarget detection.
Multimodal Learning With Transformers: A Survey

Transformer is a promising neural network learner, and has achieved great success in various machine learning tasks. Thanks to the recent prevalence of multimodal applications and Big Data, Transformer-based multimodal learning has become a hot topic in AI research. This paper presents a comprehensive survey of Transformer techniques oriented at multimodal data. The main contents of this survey include: (1) a background of multimodal learning, Transformer ecosystem, and the multimodal Big Data era, (2) a systematic review of Vanilla Transformer, Vision Transformer, and multimodal Transformers, from a geometrically topological perspective, (3) a review of multimodal Transformer applications, via two important paradigms, i.e., for multimodal pretraining and for specific multimodal tasks, (4) a summary of the common challenges and designs shared by the multimodal Transformer models and applications, and (5) a discussion of open problems and potential research directions for the community.
SF-FWA: A Self-Adaptive Fast Fireworks Algorithm for effective large-scale optimization

Computationally efficient algorithms for large-scale black-box optimization have become increasingly important in recent years due to the growing complexity of engineering and scientific problems. In this paper, a novel algorithm called the Self-adaptive Fast Fireworks Algorithm (SF-FWA) is proposed to effectively conduct large-scale black-box optimization. The main idea is to utilize a set of expressive and computationally efficient search distributions to cope with different function landscapes while tuning the hyperparameters of the search distributions in an online fashion. To achieve this, the Expressive Fast Explosion (EFE) mechanism is designed to achieve effective and efficient sampling, and the Inter-Fireworks Competitive Cooperation (IFCC) mechanism is designed to adapt hyperparameter distributions. This new optimization paradigm equips the population with the ability to automatically adjust to a rich set of function landscapes with linear computational complexity in terms of problem dimensionality. Experimental studies show that SF-FWA can not only exploit the separability of the problem efficiently but can also deal with rotational transformations to the coordinate system. The numerical results on the standard large-scale optimization benchmark suite indicate that SF-FWA outperforms current state-of-the-art large-scale optimization algorithms. The outstanding performance of SF-FWA on optimizing neural network controllers for solving reinforcement learning tasks demonstrates its great potential to be applied to a wider range of real-world problems.