Reviewerly/exHarmony huggingface dataset:
End-to-End Multi-Person Pose Estimation with Transformers

Current methods of multi-person pose estimation typically treat the localization and association of body joints separately. In this paper, we propose the first fully end-to-end multi-person Pose Estimation framework with TRansformers, termed PETR. Our method views pose estimation as a hierarchical set prediction problem and effectively removes the need for many hand-crafted modules like RoI cropping, NMS and grouping post-processing. In PETR, multiple pose queries are learned to directly reason a set of full-body poses. Then a joint decoder is utilized to further refine the poses by exploring the kinematic relations between body joints. With the attention mechanism, the proposed method is able to adaptively attend to the features most relevant to target keypoints, which largely overcomes the feature misalignment difficulty in pose estimation and improves the performance considerably. Extensive experiments on the MS COCO and CrowdPose benchmarks show that PETR plays favorably against state-of-the-art approaches in terms of both accuracy and efficiency. The code and models are available at https://github.com/hikvision-research/opera.
Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization

Unsupervised localization and segmentation are long-standing computer vision challenges that involve decom-posing an image into semantically meaningful segments without any labeled data. These tasks are particularly interesting in an unsupervised setting due to the difficulty and cost of obtaining dense image annotations, but existing un-supervised approaches struggle with complex scenes containing multiple objects. Differently from existing methods, which are purely based on deep learning, we take inspiration from traditional spectral segmentation methods by re-framing image decomposition as a graph partitioning problem. Specifically, we examine the eigenvectors of the Laplacian of a feature affinity matrix from self-supervised networks. We find that these eigenvectors already decompose an image into meaningful segments, and can be readily used to localize objects in a scene. Furthermore, by clustering the features associated with these segments across a dataset, we can obtain well-delineated, nameable regions, i.e. semantic segmentations. Experiments on complex datasets (PASCAL VOC, MS-COCO) demonstrate that our simple spectral method outperforms the state-of-the-art in unsupervised localization and segmentation by a significant margin. Furthermore, our method can be readily usedfor a variety of complex image editing tasks, such as background removal and compositing. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Project Page: https://lukemelas.github.io/deep-spectral-segmentation/
Growth Optimizer: A powerful metaheuristic algorithm for solving continuous and discrete global optimization problems

In this paper, a novel and powerful metaheuristic optimizer, named the growth optimizer (GO), is proposed. Its main design inspiration originates from the learning and reflection mechanisms of individuals in their growth processes in society. Learning is the process of individuals growing up by acquiring knowledge from the outside world. Reflection is the process of checking the individual’s own deficiencies and adjusting the individual’s learning strategies to help the individual’s growth. This work simulates this growth behavior mathematically and benchmarks the proposed algorithm on a total of 30 international test functions of the 2017 IEEE Congress on Evolutionary Computation real-parameter boundary constraint benchmark (CEC 2017 test suite). A total of 50 state-of-the-art metaheuristic algorithms participated in the comparison process. The results of the convergence accuracy comparison and the two nonparametric statistics based on the Friedman test and the Wilcoxon signed-rank test showed that GO provides competitive results compared to the 50 state-of-the-art metaheuristic algorithms tested. In addition, to verify that GO has the ability to solve different real-world optimization problems, GO was applied to two different types of real-world optimization problems: the multiple sequence alignment (MSA) problem based on the hidden Markov model (HMM) and the multithresholding image segmentation problem based on Kapur’s entropy method. GO provided more promising results than other metaheuristic techniques, especially in terms of solution quality and avoidance of local optima. The source code of the GO algorithm is publicly available at https://github.com/tsingke/Growth-Optimizer.
A novel combined approach based on deep Autoencoder and deep classifiers for credit card fraud detection

Due to the growth of e-commerce and online payment methods, the number of fraudulent transactions has increased. Financial institutions with online payment systems must utilize automatic fraud detection systems to reduce losses incurred due to fraudulent activities. The problem of fraud detection is often formulated as a binary classification model that can distinguish fraudulent transactions. Embedding the input data of the fraud dataset into a lower-dimensional representation is crucial to building robust and accurate fraud detection systems. This study proposes a two-stage framework to detect fraudulent transactions that incorporates a deep Autoencoder as a representation learning method, and supervised deep learning techniques. The experimental evaluations revealed that the proposed approach improves the performance of the employed deep learning-based classifiers. Specifically, the utilized deep learning classifiers trained on the transformed data set obtained by the deep Autoencoder significantly outperform their baseline classifiers trained on the original data in terms of all performance measures. Besides, models created using deep Autoencoder outperform those created using the principal component analysis (PCA)-obtained dataset as well as the existing models.
GBH-YOLOv5: Ghost Convolution with BottleneckCSP and Tiny Target Prediction Head Incorporating YOLOv5 for PV Panel Defect Detection

Photovoltaic (PV) panel surface-defect detection technology is crucial for the PV industry to perform smart maintenance. Using computer vision technology to detect PV panel surface defects can ensure better accuracy while reducing the workload of traditional worker field inspections. However, multiple tiny defects on the PV panel surface and the high similarity between different defects make it challenging to accurately identify and detect such defects. This paper proposes an approach named Ghost convolution with BottleneckCSP and a tiny target prediction head incorporating YOLOv5 (GBH-YOLOv5) for PV panel defect detection. To ensure better accuracy on multiscale targets, the BottleneckCSP module is introduced to add a prediction head for tiny target detection to alleviate tiny defect misses, using Ghost convolution to improve the model inference speed and reduce the number of parameters. First, the original image is compressed and cropped to enlarge the defect size physically. Then, the processed images are input into GBH-YOLOv5, and the depth features are extracted through network processing based on Ghost convolution, the application of the BottleneckCSP module, and the prediction head of tiny targets. Finally, the extracted features are classified by a Feature Pyramid Network (FPN) and a Path Aggregation Network (PAN) structure. Meanwhile, we compare our method with state-of-the-art methods to verify the effectiveness of the proposed method. The proposed PV panel surface-defect detection network improves the mAP performance by at least 27.8%.
Deep reinforcement learning in recommender systems: A survey and new perspectives

In light of the emergence of deep reinforcement learning (DRL) in recommender systems research and several fruitful results in recent years, this survey aims to provide a timely and comprehensive overview of recent trends of deep reinforcement learning in recommender systems. We start by motivating the application of DRL in recommender systems, followed by a taxonomy of current DRL-based recommender systems and a summary of existing methods. We discuss emerging topics, open issues, and provide our perspective on advancing the domain. The survey serves as introductory material for readers from academia and industry to the topic and identifies notable opportunities for further research.
Lightweight-BIoV: Blockchain Distributed Ledger Technology (BDLT) for Internet of Vehicles (IoVs)

The vast enhancement in the development of the Internet of Vehicles (IoV) is due to the impact of the distributed emerging technology and topology of the industrial IoV. It has created a new paradigm, such as the security-related resource constraints of Industry 5.0. A new revolution and dimension in the IoV popup raise various critical challenges in the existing information preservation, especially in node transactions and communication, transmission, trust and privacy, and security-protection-related problems, which have been analyzed. These aspects pose serious problems for the industry to provide vehicular-related data integrity, availability, information exchange reliability, provenance, and trustworthiness for the overall activities and service delivery prospects against the increasing number of multiple transactions. In addition, there has been a lot of research interest that intersects with blockchain and Internet of Vehicles association. In this regard, the inadequate performance of the Internet of Vehicles and connected nodes and the high resource requirements of the consortium blockchain ledger have not yet been tackled with a complete solution. The introduction of the NuCypher Re-encryption infrastructure, hashing tree and allocation, and blockchain proof-of-work require more computational power as well. This paper contributes in two different folds. First, it proposes a blockchain sawtooth-enabled modular architecture for protected, secure, and trusted execution, service delivery, and acknowledgment with immutable ledger storage and security and peer-to-peer (P2P) network on-chain and off-chain inter-communication for vehicular activities. Secondly, we design and create a smart contract-enabled data structure in order to provide smooth industrial node streamlined transactions and broadcast content. Substantially, we develop and deploy a hyperledger sawtooth-aware customized consensus for multiple proof-of-work investigations. For validation purposes, we simulate the exchange of information and related details between connected devices on the IoV. The simulation results show that the proposed architecture of BIoV reduces the cost of computational power down to 37.21% and the robust node generation and exchange up to 56.33%. Therefore, only 41.93% and 47.31% of the Internet of Vehicles-related resources and network constraints are kept and used, respectively.
A population state evaluation-based improvement framework for differential evolution

Differential evolution (DE) is one of the most efficient evolutionary algorithms for solving numerical optimization problems; however, it still suffers from premature convergence and stagnation. To address these problems, we propose a population state evaluation (PSE)-based improvement framework that can be freely embedded into various existing DE variants and population-based metaheuristic algorithms. The PSE framework comprises two population state evaluation mechanisms: one for tracking the optimization state of the population during evolution and the other for evaluating the distribution state of individuals in the population to identify the specific problem (premature convergence or stagnation) encountered by the corresponding algorithm. In addition, we design two intervention operations (dispersion and aggregation) to address premature convergence and stagnation. To verify the effectiveness of the PSE framework, we conduct comparison experiments using nine algorithms (including two basic DE algorithms, six state-of-the-art DE variants, and one non-DE algorithm) to optimize four real-world problems and 59 test functions from the IEEE CEC 2014 and IEEE CEC 2017 testbeds. The experimental results show that the PSE framework can significantly improve the optimization performance of the existing algorithms.
STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition

In action recognition, although the combination of spatiotemporal videos and skeleton features can improve the recognition performance, a separate model and balancing feature representation for cross-modal data are required. To solve these problems, we propose Spatio-TemporAl cRoss (STAR)-transformer, which can effectively represent two cross-modal features as a recognizable vector. First, from the input video and skeleton sequence, video frames are output as global grid tokens and skeletons are output as joint map tokens, respectively. These tokens are then aggregated into multi-class tokens and input into STAR-transformer. The STAR-transformer encoder consists of a full spatio-temporal attention (FAttn) module and a proposed zigzag spatio-temporal attention (ZAttn) module. Similarly, the continuous decoder consists of a FAttn module and a proposed binary spatio-temporal attention (BAttn) module. STAR-transformer learns an efficient multi-feature representation of the spatio-temporal features by properly arranging pairings of the FAttn, ZAttn, and BAttn modules. Experimental results on the Penn-Action, NTU-RGB+D 60, and 120 datasets show that the proposed method achieves a promising improvement in performance in comparison to previous state-of-the-art methods.
Large Language Models and the Reverse Turing Test

Large Language Models (LLMs) have been transformative. They are pre-trained foundational models that are self-supervised and can be adapted with fine tuning to a wide range of natural language tasks, each of which previously would have required a separate network model. This is one step closer to the extraordinary versatility of human language. GPT-3 and more recently LaMDA can carry on dialogs with humans on many topics after minimal priming with a few examples. However, there has been a wide range of reactions and debate on whether these LLMs understand what they are saying or exhibit signs of intelligence. This high variance is exhibited in three interviews with LLMs reaching wildly different conclusions. A new possibility was uncovered that could explain this divergence. What appears to be intelligence in LLMs may in fact be a mirror that reflects the intelligence of the interviewer, a remarkable twist that could be considered a Reverse Turing Test. If so, then by studying interviews we may be learning more about the intelligence and beliefs of the interviewer than the intelligence of the LLMs. As LLMs become more capable they may transform the way we interact with machines and how they interact with each other. Increasingly, LLMs are being coupled with sensorimotor devices. LLMs can talk the talk, but can they walk the walk? A road map for achieving artificial general autonomy is outlined with seven major improvements inspired by brain systems. LLMs could be used to uncover new insights into brain function by downloading brain data during natural behaviors.
A comprehensive survey on design and application of autoencoder in deep learning

Autoencoder is an unsupervised learning model, which can automatically learn data features from a large number of samples and can act as a dimensionality reduction method. With the development of deep learning technology, autoencoder has attracted the attention of many scholars. Researchers have proposed several improved versions of autoencoder based on different application fields. First, this paper explains the principle of a conventional autoencoder and investigates the primary development process of an autoencoder. Second, We proposed a taxonomy of autoencoders according to their structures and principles. The related autoencoder models are comprehensively analyzed and discussed. This paper introduces the application progress of autoencoders in different fields, such as image classification and natural language processing, etc. Finally, the shortcomings of the current autoencoder algorithm are summarized, and prospected for its future development directions are addressed.
Immersive time (ImT): Conceptualizing time spent in the metaverse

With growing investment into the metaverse or metaverses, the required hardware and software is becoming more powerful and cheaper, and tech firms' expectations for this market are high. In parallel, consumer brands are claiming their digital real estate as consumers seek experiences beyond those available in the real world. Having people engage on this platform is important for all stakeholders. To address this much needed level of engagement, this editorial introduces the concept of immersive time (ImT), which is described as the conscious, deliberate, and dedicated time spent using a headset and other accessories to continually engage in the metaverse, presumably at least in part to escape the real world. Once in ImT, there are rather limited opportunities to multitask and simultaneously devote time to the activities within the metaverse and the real world. As such, while prolonged ImT is important for the stakeholders, it is imperative to recognize the inherent dark sides of this time. Understanding ImT has important implications for researchers and practitioners (including tech developers, and brand and marketing managers who contemplate their metaverse strategies), and policymakers who aim to ensure customers' safety when immersing in the metaverse. This editorial concludes with an agenda for future research to enhance our understanding of consumer behaviors and engagement in the metaverse.
ImageBind One Embedding Space to Bind Them All

We present ImageBind, an approach to learn a joint embedding across six different modalities - images, text, audio, depth, thermal, and IMU data. We show that all combinations of paired data are not necessary to train such a joint embedding, and only image-paired data is sufficient to bind the modalities together. ImageBind can leverage recent large scale vision-language models, and extends their zero-shot capabilities to new modalities just by using their natural pairing with images. It enables novel emergent applications ‘out-of-the-box’ including cross-modal retrieval, composing modalities with arithmetic, cross-modal detection and generation. The emergent capabilities improve with the strength of the image encoder and we set a new state-of-the-art on emergent zero-shot recognition tasks across modalities, outperforming specialist supervised models. Finally, we show strong few-shot recognition results outperforming prior work, and that ImageBind serves as a new way to evaluate vision models for visual and non-visual tasks.
InstructPix2Pix: Learning to Follow Image Editing Instructions

We propose a method for editing images from human instructions: given an input image and a written instruction that tells the model what to do, our model follows these instructions to edit the image. To obtain training data for this problem, we combine the knowledge of two large pretrained models—a language model (GPT-3) and a text-to-image model (Stable Diffusion)—to generate a large dataset of image editing examples. Our conditional diffusion model, InstructPix2Pix, is trained on our generated data, and generalizes to real images and user-written instructions at inference time. Since it performs edits in the forward pass and does not require per-example fine-tuning or inversion, our model edits images quickly, in a matter of seconds. We show compelling editing results for a diverse collection of input images and written instructions.
Context-Aware Mouse Behavior Recognition Using Hidden Markov Models

Automated recognition of mouse behaviors is crucial in studying psychiatric and neurologic diseases. To achieve this objective, it is very important to analyze the temporal dynamics of mouse behaviors. In particular, the change between mouse neighboring actions is swift in a short period. In this paper, we develop and implement a novel hidden Markov model (HMM) algorithm to describe the temporal characteristics of mouse behaviors. In particular, we here propose a hybrid deep learning architecture, where the first unsupervised layer relies on an advanced spatial-temporal segment Fisher vector encoding both visual and contextual features. Subsequent supervised layers based on our segment aggregate network are trained to estimate the state-dependent observation probabilities of the HMM. The proposed architecture shows the ability to discriminate between visually similar behaviors and results in high recognition rates with the strength of processing imbalanced mouse behavior datasets. Finally, we evaluate our approach using JHuang's and our own datasets, and the results show that our method outperforms other state-of-the-art approaches.
An Improved Squirrel Search Algorithm for Global Function Optimization

An improved squirrel search algorithm (ISSA) is proposed in this paper. The proposed algorithm contains two searching methods, one is the jumping search method, and the other is the progressive search method. The practical method used in the evolutionary process is selected automatically through the linear regression selection strategy, which enhances the robustness of squirrel search algorithm (SSA). For the jumping search method, the ‘escape’ operation develops the search space sufficiently and the ‘death’ operation further explores the developed space, which balances the development and exploration ability of SSA. Concerning the progressive search method, the mutation operation fully preserves the current evolutionary information and pays more attention to maintain the population diversity. Twenty-one benchmark functions are selected to test the performance of ISSA. The experimental results show that the proposed algorithm can improve the convergence accuracy, accelerate the convergence speed as well as maintain the population diversity. The statistical test proves that ISSA has significant advantages compared with SSA. Furthermore, compared with five other intelligence evolutionary algorithms, the experimental results and statistical tests also show that ISSA has obvious advantages on convergence accuracy, convergence speed and robustness.
Coherent Comments Generation for Chinese Articles with a Graph-to-Sequence Model

Automatic article commenting is helpful in encouraging user engagement and interaction on online news platforms.However, the news documents are usually too long for traditional encoder-decoder based models, which often results in general and irrelevant comments.In this paper, we propose to generate comments with a graph-to-sequence model that models the input news as a topic interaction graph.By organizing the article into graph structure, our model can better understand the internal structure of the article and the connection between topics, which makes it better able to understand the story.We collect and release a large scale news-comment corpus from a popular Chinese online news platform Tencent Kuaibao. 1 Extensive experiment results show that our model can generate much more coherent and informative comments compared with several strong baseline models. 2