Reviewerly/exHarmony huggingface dataset:
Injecting Semantic Concepts into End-to-End Image Captioning

Tremendous progresses have been made in recent years in developing better image captioning models, yet most of them rely on a separate object detector to extract regional features. Recent vision-language studies are shifting towards the detector-free trend by leveraging grid representations for more flexible model training and faster inference speed. However, such development is primarily focused on image understanding tasks, and remains less investigated for the caption generation task. In this paper, we are concerned with a better-performing detector-free image captioning model, and propose a pure vision transformer-based image captioning model, dubbed as ViTCAP, in which grid representations are used without extracting the regional features. For improved performance, we introduce a novel Concept Token Network (CTN) to predict the semantic concepts and then incorporate them into the end-to-end captioning. In particular, the CTN is built on the basis of a vision transformer, and is designed to predict the concept tokens through a classification task, from which the rich semantic information contained greatly benefits the captioning task. Compared with the previous detector-based models, ViTCAP drastically simplifies the architectures and at the same time achieves competitive performance on various challenging image captioning datasets. In particular, ViTCAP reaches 138.1 CIDEr scores on COCO-caption Karpathy-split, 93.8 and 108.6 CIDEr scores on nocaps and Google-CC captioning datasets, respectively.
Blockchain-Based Two-Stage Federated Learning With Non-IID Data in IoMT System

The Internet of Medical Things (IoMT) has a bright future with the development of smart mobile devices. Information technology is also leading changes in the healthcare industry. IoMT devices can detect patient signs and provide treatment guidance and even instant diagnoses through technologies, such as artificial intelligence (AI) and wireless communication. However, conventional centralized machine learning approaches are often difficult to apply within IoMT devices because of the difficulty of large-scale collection of patient data and the potential risk of privacy breaches. Therefore, we propose a blockchain-based two-stage federated learning approach that allows IoMT devices to train a global model collaboratively without gathering the data to a central server. Specifically, to address the problem of poor training performance on non-independent identically distributed (non-IID) data, we design a blockchain-based data-sharing scheme that can significantly improve the model’s accuracy without threatening user privacy. We also design a client selection mechanism to further improve the system’s efficiency. Finally, we validate the feasibility and effectiveness of our system through simulation experiments on three popular datasets (i.e., MNIST, Fashion-MNIST, and CIFAR-10).
Align and Prompt: Video-and-Language Pre-training with Entity Prompts

Yidco-and-language pre-training has shown promising improvements on various downstream tasks. Most previous methods capture cross-modal interactions with a standard transformer-based multimodal encoder, not fully addressing the misalignment between unimodal video and text features. Besides, learning finegrained visual-language alignment usually requires off-the-shelf object detectors to provide object information, which is bottlenecked by the detector's limited vocabulary and expensive computation cost. In this paper, we propose Align and Prompt: a new video-and-language pre-training framework (AlPro), which operates on sparsely-sampled video frames and achieves more effective cross-modal alignment without explicit object detectors. First, we introduce a video-text contrastive (VTC) loss to align unimodal video-text features at the instance level, which eases the modeling of cross-modal interactions. Then, we propose a novel visually-grounded pre-training task, prompting entity modeling (PEM), which learns finegrained alignment between visual region and text entity via an entity prompter module in a self-supervised way. Finally, we pretrain the video-and-language transformer models on large webly-source video-text pairs using the proposed VTC and PEM losses as well as two standard losses of masked language modeling (MLM) and video-text matching (VTM). The resulting pre-trained model achieves state-of-the-art performance on both text-video retrieval and videoQA, outperforming prior work by a substantial margin. Implementation and pre-trained models are available at https://github.com/salesforce/ALPRO.
Secure image encryption scheme based on a new robust chaotic map and strong S-box

Because the characteristics of chaotic systems are very consistent with the requirements of cryptosystem, chaotic systems are widely used in the encryption of big data, especially the chaos based image encryption is particularly feasible. In this paper, a one-dimensional (1D) piece-wise quadratic polynomial chaotic map (PWQPCM) is proposed to obtain a new system model with better chaotic characteristics. Compared with some existing 1D chaotic systems, the proposed PWQPCM system has better dynamic performance. In particular, it exhibits robust chaotic behavior over a larger continuous parameter range, which is more suitable for cryptographic applications. In addition, a strong S-box construction method and a secure image encryption algorithm based on PWQPCM model are proposed. In this encryption algorithm, a new method of pixel segmentation encryption, S-box substitution and diffusion encryption is combined. The cryptosystem proposed in this paper has the advantages of good performance against data loss, low time cost and flexible adjustment of security strength. Simulation experiments and security analysis verified the above good performance of the encryption system, showing its good application potential in real-time image security communication.
Modified Genetic Algorithm with Deep Learning for Fraud Transactions of Ethereum Smart Contract

Recently, the Ethereum smart contracts have seen a surge in interest from the scientific community and new commercial uses. However, as online trade expands, other fraudulent practices—including phishing, bribery, and money laundering—emerge as significant challenges to trade security. This study is useful for reliably detecting fraudulent transactions; this work developed a deep learning model using a unique metaheuristic optimization strategy. The new optimization method to overcome the challenges, Optimized Genetic Algorithm-Cuckoo Search (GA-CS), is combined with deep learning. In this research, a Genetic Algorithm (GA) is used in the phase of exploration in the Cuckoo Search (CS) technique to address a deficiency in CS. A comprehensive experiment was conducted to appraise the efficiency and performance of the suggested strategies compared with those of various popular techniques, such as k-nearest neighbors (KNN), logistic regression (LR), multi-layer perceptron (MLP), XGBoost, light gradient boosting machine (LGBM), random forest (RF), and support vector classification (SVC), in terms of restricted features and we compared their performance and efficiency metrics to the suggested approach in detecting fraudulent behavior on Ethereum. The suggested technique and SVC models outperform the rest of the models, with the highest accuracy, while deep learning with the proposed optimization strategy outperforms the RF model, with slightly higher performance of 99.71% versus 98.33%.
Pond: CXL-Based Memory Pooling Systems for Cloud Platforms

Public cloud providers seek to meet stringent performance requirements and low hardware cost. A key driver of performance and cost is main memory. Memory pooling promises to improve DRAM utilization and thereby reduce costs. However, pooling is challenging under cloud performance requirements. This paper proposes Pond, the first memory pooling system that both meets cloud performance goals and significantly reduces DRAM cost. Pond builds on the Compute Express Link (CXL) standard for load/store access to pool memory and two key insights. First, our analysis of cloud production traces shows that pooling across 8-16 sockets is enough to achieve most of the benefits. This enables a small-pool design with low access latency. Second, it is possible to create machine learning models that can accurately predict how much local and pool memory to allocate to a virtual machine (VM) to resemble same-NUMA-node memory performance. Our evaluation with 158 workloads shows that Pond reduces DRAM costs by 7% with performance within 1-5% of same-NUMA-node VM allocations.
Joint Secure Offloading and Resource Allocation for Vehicular Edge Computing Network: A Multi-Agent Deep Reinforcement Learning Approach

The mobile edge computing (MEC) technology can simultaneously provide high-speed computing services for multiple vehicular users (VUs) in vehicular edge computing (VEC) networks. Nevertheless, due to the open feature of the wireless offloading channels and the high mobility of the vehicles, the security and stability of the offloading process would be seriously degraded. In this paper, by utilizing the physical layer security (PLS) technique and spectrum sharing architecture, we propose a deep reinforcement learning based joint secure offloading and resource allocation (SORA) scheme to improve the secrecy performance and resource efficiency of the multi-user VEC networks, where the VU offloading links share the frequency spectrum preoccupied with the vehicle-to-vehicle (V2V) communication links. We use Wyner’s wiretap coding scheme to obtain the achievable secrecy rate and guarantee that confidential information cannot be decoded by multiple mobile eavesdroppers. We aim at minimizing the system processing delay while securing the wireless offloading process, by jointly optimizing the transmit power, the frequency spectrum selection and the computation resource allocation. We formulate the optimization problem as a multi-agent collaborative optimal decision problem and solve it with a double deep Q-learning algorithm. Besides, we set a punishment mechanism for the rate degradation to guarantee the communication quality of each V2V link. Simulation results demonstrate that multiple VU agents adopting the SORA scheme can rapidly adapt to the highly dynamic VEC networks and cooperate to improve the system delay performance while increasing the secrecy probability.
Review of Autonomous Path Planning Algorithms for Mobile Robots

Mobile robots, including ground robots, underwater robots, and unmanned aerial vehicles, play an increasingly important role in people’s work and lives. Path planning and obstacle avoidance are the core technologies for achieving autonomy in mobile robots, and they will determine the application prospects of mobile robots. This paper introduces path planning and obstacle avoidance methods for mobile robots to provide a reference for researchers in this field. In addition, it comprehensively summarizes the recent progress and breakthroughs of mobile robots in the field of path planning and discusses future directions worthy of research in this field. We focus on the path planning algorithm of a mobile robot. We divide the path planning methods of mobile robots into the following categories: graph-based search, heuristic intelligence, local obstacle avoidance, artificial intelligence, sampling-based, planner-based, constraint problem satisfaction-based, and other algorithms. In addition, we review a path planning algorithm for multi-robot systems and different robots. We describe the basic principles of each method and highlight the most relevant studies. We also provide an in-depth discussion and comparison of path planning algorithms. Finally, we propose potential research directions in this field that are worth studying in the future.
Very-large-scale integrated quantum graph photonics

Abstract Graphs have provided an expressive mathematical tool to model quantum-mechanical devices and systems. In particular, it has been recently discovered that graph theory can be used to describe and design quantum components, devices, setups and systems, based on the two-dimensional lattice of parametric nonlinear optical crystals and linear optical circuits, different to the standard quantum photonic framework. Realizing such graph-theoretical quantum photonic hardware, however, remains extremely challenging experimentally using conventional technologies. Here we demonstrate a graph-theoretical programmable quantum photonic device in very-large-scale integrated nanophotonic circuits. The device monolithically integrates about 2,500 components, constructing a synthetic lattice of nonlinear photon-pair waveguide sources and linear optical waveguide circuits, and it is fabricated on an eight-inch silicon-on-insulator wafer by complementary metal–oxide–semiconductor processes. We reconfigure the quantum device to realize and process complex-weighted graphs with different topologies and to implement different tasks associated with the perfect matching property of graphs. As two non-trivial examples, we show the generation of genuine multipartite multidimensional quantum entanglement with different entanglement structures, and the measurement of probability distributions proportional to the modulus-squared hafnian (permanent) of the graph’s adjacency matrices. This work realizes a prototype of graph-theoretical quantum photonic devices manufactured by very-large-scale integration technologies, featuring arbitrary programmability, high architectural modularity and massive manufacturing scalability.
Decentralized P2P Federated Learning for Privacy-Preserving and Resilient Mobile Robotic Systems

Swarms of mobile robots are being widely applied for complex tasks in various practical scenarios toward modern smart industry. Federated learning (FL) has been developed as a promising privacy-preserving paradigm to tackle distributed machine learning tasks for mobile robotic systems in 5G and beyond networks. However, unstable wireless network conditions of the complex and harsh working environment may lead to poor communication quality and bring big challenges to traditional centralized global training in FL models. In this article, a Peer-to-Peer (P2P) based Privacy-Perceiving Asynchronous Federated Learning (PPAFL) framework is introduced to realize the decentralized model training for secure and resilient modern mobile robotic systems in 5G and beyond networks. Specifically, a reputation-aware coordination mechanism is designed and addressed to coordinate a group of smart devices dynamically into a virtual cluster, in which the asynchronous model aggregation is conducted in a decentralized P2P manner. A secret sharing based communication mechanism is developed to ensure an encrypted P2P FL process, while a Secure Stochastic Gradient Descent (SSGD) scheme is integrated with an Autoencoder and a Gaussian mechanism is developed to ensure an anonymized local model update, communicating within a few neighboring clients. The case study based experiment and evaluation in three different application scenarios demonstrate that the PPAFL can effectively improve the security and resilience issues compared with the traditional centralized approaches for smart mobile robotic applications in 5G and beyond networks.
Improving person re-identification by multi-task learning

For person re-identification, the core task is to find effective representations of a person image. As Multi-Task Learning can achieve great performance in seeking robust features, we propose a novel Multi-Task Learning Network (MTNet) with four different losses for person re-identification (re-ID). Our MTNet is an end-to-end deep learning framework, which all the parameters and losses can be jointly optimized. In our method we combine two tasks closely corresponding to person re-identification, pedestrian identity task and pedestrian attribute task, who provide complementary information from different perspective by integrating multi-context informations. Attribute focuses on some special aspects of a person, while identity pays more attention to overall contour and appearance. Meanwhile, both classification and verification losses are employed to optimize the distance of samples. Identification losses are used to construct a large class space, while verification losses are applied optimize the space by minimizing the distance between similar images and maximizing the distance between dissimilar images. In the experiments, our MTNet achieves the state-of-the-art results on two typical datasets Market1501 [1] and DukeMTMC-reID [2].
Multi-image encryption scheme with quaternion discrete fractional Tchebyshev moment transform and cross-coupling operation

With quaternion theory, the traditional discrete fractional Tchebyshev transform is extended to the quaternion algebra domain for multi-image processing. A new multi-image encryption scheme based on quaternion discrete fractional Tchebyshev moment transform (QDFrTMT) and the cross-coupling chaotic system is suggested. The original images are first confused by fractal sorting square matrix and sine-logistic exponential chaotic map, and then the resulting image is divided into four matrices. With the quaternion symplectic form representation, a quaternion signal can be formed by the divided matrices, which can reduce the number of the discrete fractional Tchebyshev transforms used and enhance the computational efficiency. Subsequently, the quaternion array is encrypted with the proposed QDFrTMT. To overcome the weakness of the permutation-diffusion operation based on a single chaotic map, a Logistic-sine exponential chaotic map and a piece-wise linear chaotic map are cross-coupled. The final ciphertext images can be acquired by carrying out the dual-layer encryption processes in horizontal and vertical directions. Numerical simulations and security analyses verify the effectiveness of the multi-image encryption algorithm and its strong ability to counteract common attacks.
GPT understands, too

Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance—e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.
Generative Agents: Interactive Simulacra of Human Behavior

Believable proxies of human behavior can empower interactive applications ranging from immersive environments to rehearsal spaces for interpersonal communication to prototyping tools. In this paper, we introduce generative agents: computational software agents that simulate believable human behavior. Generative agents wake up, cook breakfast, and head to work; artists paint, while authors write; they form opinions, notice each other, and initiate conversations; they remember and reflect on days past as they plan the next day. To enable generative agents, we describe an architecture that extends a large language model to store a complete record of the agent’s experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims, where end users can interact with a small town of twenty-five agents using natural language. In an evaluation, these generative agents produce believable individual and emergent social behaviors. For example, starting with only a single user-specified notion that one agent wants to throw a Valentine’s Day party, the agents autonomously spread invitations to the party over the next two days, make new acquaintances, ask each other out on dates to the party, and coordinate to show up for the party together at the right time. We demonstrate through ablation that the components of our agent architecture—observation, planning, and reflection—each contribute critically to the believability of agent behavior. By fusing large language models with computational interactive agents, this work introduces architectural and interaction patterns for enabling believable simulations of human behavior.
Longitudinally Guided Super-Resolution of Neonatal Brain Magnetic Resonance Images

Neonatal magnetic resonance (MR) images typically have low spatial resolution and insufficient tissue contrast. Interpolation methods are commonly used to upsample the images for the subsequent analysis. However, the resulting images are often blurry and susceptible to partial volume effects. In this paper, we propose a novel longitudinally guided super-resolution (SR) algorithm for neonatal images. This is motivated by the fact that anatomical structures evolve slowly and smoothly as the brain develops after birth. We propose a strategy involving longitudinal regularization, similar to bilateral filtering, in combination with low-rank and total variation constraints to solve the ill-posed inverse problem associated with image SR. Experimental results on neonatal MR images demonstrate that the proposed algorithm recovers clear structural details and outperforms state-of-the-art methods both qualitatively and quantitatively.
Distributed correlation-based feature selection in spark

Feature selection (FS) is a key preprocessing step in data mining. CFS (Correlation-Based Feature Selection) is an FS algorithm that has been successfully applied to classification problems in many domains. We describe Distributed CFS (DiCFS) as a completely redesigned, scalable, parallel and distributed version of the CFS algorithm, capable of dealing with the large volumes of data typical of big data applications. Two versions of the algorithm were implemented and compared using the Apache Spark cluster computing model, currently gaining popularity due to its much faster processing times than Hadoop's MapReduce model. We tested our algorithms on four publicly available datasets, each consisting of a large number of instances and two also consisting of a large number of features. The results show that our algorithms were superior in terms of both time-efficiency and scalability. In leveraging a computer cluster, they were able to handle larger datasets than the non-distributed WEKA version while maintaining the quality of the results, i.e., exactly the same features were returned by our algorithms when compared to the original algorithm available in WEKA.