Reviewerly/exHarmony huggingface dataset:
Lightweight image super-resolution for IoT devices using deep residual feature distillation network

The 5th industrial revolution is characterized by an extensive interconnection of embedded devices, which offer a range of services, including the monitoring of their environments. Images captured from remote cameras require enhancements for effective analysis. Despite recent progress in single-image super-resolution techniques by yielding impressive results through deep convolutional neural networks, the complexity of these advanced models renders them impractical for use on miniaturized Internet of Things (IoT) devices, primarily due to their limited computational capabilities and memory constraints. Furthermore, the rapid evolution of IoT devices necessitates efficient image super-resolution techniques, while existing advanced methods, based on deep convolutional neural networks, are too resource-intensive for these devices, and this gap highlights the need for a more suitable solution. In this study, we introduce a lightweight, efficient super-resolution model specifically designed for IoT devices. This model incorporates a novel deep residual feature distillation block (DRFDB), which leverages a depthwise-separable convolution block (DCB) for effective feature extraction. The focus is on reducing computational and memory demands without compromising on image quality. The proposed DCB extracts coarse features from given input features as calculation units, using two operations, depthwise and pointwise convolutions. These two operations are able to significantly reduce the number of parameters and floating-point operations while maintaining a PSNR value higher than the 90% threshold. We modify the proposed DCB and introduce a multi-kernel depthwise-separable convolution block (MKDCB) to fine-tune the model. The experiments, conduct on various standard datasets such as DIV2K, Set5, Set14, Urban100, and Manga109 by demonstrating that our model significantly outperforms existing methods in terms of both image quality and computational efficiency. The model shows improved performance metrics like PSNR, while requiring fewer parameters and less memory usage, making it highly suitable for IoT applications. This study presents a breakthrough in super-resolution for IoT devices, balancing high-quality image reconstruction with the limited resources of these devices.
Class incremental learning via dynamic regeneration with task-adaptive distillation

Class Incremental Learning (CIL) is a paradigm that excels in efficiently training models on an expanding set of classes, while preserving performance in classes learned earlier. In the context of smart city development, CIL plays a crucial role in addressing the need for Networking Systems of Artificial Intelligence (NSAI) to continuously adapt to ever-evolving data. In recent years, the incremental network structure widely adopted in CIL suffers from poor knowledge distillation and rapid model parameter growth. In this paper, we introduce a novel strategy named dynamic regeneration with task-adaptive distillation (DRTAD), which dynamically adapts to new tasks, and sustains good performance even as the class set continues to grow. DRTAD adopts a two-stage training strategy: dynamic regeneration and dynamic retention. During dynamic regeneration, DRTAD enhances feature representation by creating a new feature extraction module that extracts features from new classes, while also utilizing features from previously learned classes. Additionally, DRTAD introduces task-adaptive distillation to improve the poor knowledge distillation, further mitigating catastrophic forgetting. During the dynamic retention phase, DRTAD achieves a higher pruning rate through RM operation. Comprehensive experiments on CIFAR-100 and ImageNet-100 datasets demonstrate DRTAD’s superior performance compared to existing CIL methods. Notably, in the CIFAR100-B50 5 steps incremental setting, DRTAD increases the last-phase accuracy from 65.51% to 70.54% (＋5.03%), while maintaining fewer parameters (−11.1%). Similarly, in the ImageNet100-B50 10 steps setting, the last-phase accuracy rises from 70.04% to 72.50% (＋2.46%). These results indicate DRTAD’s efficacy in mitigating catastrophic forgetting in incremental structure.
Multiple integration model for single-source domain generalizable person re-identification

Domain generalizable (DG) person re-identification (re-ID) aims to train a model on labeled source domains which can perform well on invisible target domains. Because of the distribution shifts between different domains, it is a challenging task. Existing methods address this challenge by using multiple source domains to train a model which requires more data, manual labor, and computation. In contrast, we pay attention to the single-source DG re-ID task, that is, only one source domain data is accessible for training. However, due to the limited availability of training data, this task is more difficult. In this paper, a novel MulTiple Integration (MTI) model is introduced for single-source DG person re-ID. By integrating multiple reliable perturbations, the generalization performance can be improved. Specifically, MTI model contains two types of integration modules, one is shallow-level compensation (SLC) and the other is deep-level integration (DLI). For SLC, according to the idea of continual learning, the shallow-level information of the ImageNet pre-trained ResNet-50 branch is introduced and fused with the shallow-level information of our backbone network. In this way, massive information in ImageNet can be used to prevent the disastrous forgetting of the pre-trained information, and information compensation can be provided for backbone network. Additionally, we propose a hybrid integrated normalization layer to fuse information and improve the model’s generalization performance. For DLI, a wave transformer block is introduced in the deep layer of the backbone, which can integrate the information of a batch images and contain reliable disturbance, so that the robustness of the model can be promoted. Extensive experimental results demonstrate the superiority of our model.
TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis

Multimodal emotion analysis is an important endeavor in human–computer interaction research, as it enables the accurate identification of an individual’s emotional state by simultaneously analyzing text, video, and sound features. Although current emotion recognition algorithms have performed well using multimodal fusion strategies, two key challenges remain. The first challenge is the efficient extraction of modality-invariant and modality-specific features prior to fusion, which requires deep feature interactions between the different modalities. The second challenge concerns the ability to distinguish high-level semantic relations between modality features. To address these issues, we propose a new modality-binding learning framework and redesign the internal structure of the transformer model. Our proposed modality binding learning model addresses the first challenge by incorporating bimodal and trimodal binding mechanisms. These mechanisms handle modality-specific and modality-invariant features, respectively, and facilitate cross-modality interactions. Furthermore, we enhance feature interactions by introducing fine-grained convolution modules in the feedforward and attention layers of the transformer structure. To address the second issue, we introduce CLS and PE feature vectors for modality-invariant and modality-specific features, respectively. We use similarity loss and dissimilarity loss to support model convergence. Experiments on the widely used MOSI and MOSEI datasets show that our proposed method outperforms state-of-the-art multimodal sentiment classification approaches, confirming its effectiveness and superiority. The source code can be found at https://github.com/JackAILab/TMBL.
Mobility and dependency-aware task offloading for intelligent assisted driving in vehicular edge computing networks

Intelligent assisted driving is an important application in vehicular edge computing networks (VECNs). In the intelligent transportation system (ITS), a group of moving vehicle users need to be coordinated to complete complex vehicular applications. A number of dependent, latency-sensitive, and computation-intensive tasks are generated. However, the existing works have given less consideration to the dependencies among both vehicle users and the subtasks in vehicle, which makes it a huge challenge to complete tasks timely. When interdependent tasks come from different vehicle users, a special task preparation time is needed, which can disrupt the ongoing task processing. Furthermore, the high mobility of vehicles directly affects the data transmission rate. To address the mentioned challenges, we design an efficient mobility and dependency-aware task offloading strategy in VECNs. The objective is to minimize both the overall system task completion delay and the economic cost. We take into account the real-time locations and task preparation time of vehicle users. Additionally, we propose a multi-decision-making offloading algorithm (MDOA) that primarily analyzes the processing priorities for both vehicle users and subtasks. In order to integrate practical applications, the financial expenses of vehicle users are also considered as an indispensable part. As a result, we propose an efficient two-step task offloading algorithm. Through numerous simulation examples, we demonstrate the efficiency and high performance of the proposed task offloading strategies in VECNs when compared to existing algorithms.
Neural ordinary differential equation for irregular human motion prediction

Human motion prediction often assumes that the input sequence is of fixed frame rates. However, in real-world applications, the motion capture system may work unstably sometimes and miss some frames, which leads to inferior performance. To solve this problem, this paper leverages neural Ordinary Differential Equations and proposes a human Motion Prediction method named MP-ODE to handle irregular-time human pose series. First, a Difference Operator and a Positional Encoding are proposed to explicitly provide the kinematic and time information for the model. Second, we construct the encoder–decoder model with ODE-GRU unit, which enables us to learn continuous-time dynamics of human motion. Third, a Quaternion Loss transforms exponential maps to quaternion to train MP-ODE. The Quaternion Loss can avoid the discontinuities and singularities of exponential maps, boosting the convergence of the model. Comprehensive experiments on Human3.6 m and CMU-Mocap datasets demonstrate that the proposed MP-ODE achieves promising performance in both normal and irregular-time conditions.
Energy-efficient resource allocation in cloud infrastructure using L3F-MGA and E-ANFIS

In industry along with academia, Task Scheduling (TS) along with Resources Allocation (RA) remains a challenging issue even though lots of work was carried out in the Cloud Computing (CC) field. CC uses massive virtualized Data Centers (DC). Still, it took an enormous quantity of energy on account of the constant demand by users for resources in addition to inefficient RA techniques. The higher Energy Consumption (EC) because of improper RA inside DC can be overcome using energy-aware RA techniques. Therefore, this paper proposes a Levy flight Firefly Modified Genetic Algorithm (L3F-MGA) as well as an Entropy-based Adaptive Neuros-Fuzzy Inferences System (E-ANFIS) for energy-efficient Resources Allocation (EE-RA) in cloud structure that manages the user demands and also lessens the EC of Cloud Data Center (CDC). In the proposed work, centred on task features together with cloud resource features, the input tasks are arranged using L3F-MGA. Conversely, the Virtual Machine (VM) status is identified. After that, centred on the task feature and VM status details, the E-ANFIS allows the active VM to complete the tasks effectively. The experiments compare the proposed method's performance for effective RA with other associated works. The proposed RA technique shows EC by 11 % lesser than the existing CM-GA. The Processing Time of E-ANFIS is 8 % lesser than CM-GA, 14 % less than ANFIS, 20 % less than GA, and 16 % less than PSO.
A survey on personalized itinerary recommendation: From optimisation to deep learning

The tourism industry is a significant contributor to the global economy, responsible for generating nearly 10% of the world’s GDP and employing around 9% of the global workforce. A crucial aspect of this industry is personalised itinerary recommendation, where visitors’ preferences and constraints are taken into account to create customised travel plans. This task involves selecting the best points of interests (POIs) for visitors in various cities and then schedule these POIs as an itinerary considering numerous constraints. However, due to the varied ways in which researchers have defined the itinerary recommendations, it can be challenging for new researchers to locate up-to-date literature on the topic. As a result, this paper aims to review existing research in this area and provide a taxonomy of the works based on problem formulations, proposed techniques, constraints, and features used. We divide the study into two directions: user satisfaction and provider satisfaction, where user satisfaction is derived non–personalised and personalised POI/ Itinerary recommendations. We also discuss the data sources, techniques ranging from optimization approaches to deep learning and evaluation methodologies commonly used in this field. Finally, we highlight the importance of personalised itinerary recommendation and identify areas for future research to address the current challenges.
Prompted representation joint contrastive learning for aspect-based sentiment analysis

As a fine-grained and challenging subtask in the natural language processing (NLP) community, aspect-based sentiment analysis (ABSA) aims to predict the sentiment polarity towards a given aspect term. In previous ABSA research, most works utilized the pre-trained language model (PLM) as the backbone of their proposed methods, without any specific task-related instructions. Besides, some works focused on learning the dependency information or the external knowledge-enhanced dependency information separately, which lacked the exploitation of the mutual interaction between the normal dependency and knowledge-enhanced dependency. Therefore, we propose a novel ABSA method namely prompted representation joint contrastive learning enhanced graph convolutional networks (PRCL-GCN) to strengthen the robustness of the ABSA model. Specifically, to achieve the task-oriented contextual representation, we design the task-specific prompt template to guide the fine-tuning process of PLM in the ABSA task. And a biaffine attention mechanism is employed to further extract the essential sentiment feature from the prompted representation. Moreover, we introduce the syntax dependency graph as prior knowledge, and construct an affective syntactic dependency graph by injecting the affective knowledge from SenticNet into the graph. Then, we utilize the multi-layer GCNs to process the above two syntactic graphs independently, which aims to learn multi-granularity syntactic features. Subsequently, a novel designed attention variant is leveraged to integrate these syntax features with the guided contextual representation, separately. Eventually, through designing a Kullback–Leibler divergence-based contrastive learning to encourage the model’s learning, we improve the model’s accuracy in modeling contextual representation by integrating the designed dual-ways information. Extensive experiments are conducted on five benchmark datasets, and the outstanding experiment results validate the effectiveness of our proposed model.
An Efficient GCN Accelerator Based on Workload Reorganization and Feature Reduction

The irregular adjacency matrix and the mismatched computation patterns of Aggregation and Combination phases make Graph Neural Networks (GNNs) challenging to compute efficiently. This paper proposes a software and hardware co-design system to reduce computational latency and memory access based on workload reorganization and feature reduction. In software, the adjacency matrix is preprocessed, and the workload in both feature and node dimensions is concentrated to optimize memory access and hardware utilization. The interlayer nodes are analyzed using Principal Component Analysis (PCA) to explore the minimum feature vector length based on information redundancy, and a unique weight initialization is utilized for retraining to trim the feature vector to the minimum length. In hardware, an efficient GCN accelerator is designed to fully support the reorganized workload by reconfigurable output node computation. The hardware accelerator is implemented using 28-nm CMOS technology. It achieves 3.3 TOPS peak throughput and 2.6 TOPS/W energy efficiency. Compared with HyGCN, this result shows that the proposed method can improve the overall performance by 5 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times$</tex-math> </inline-formula> with a negligible accuracy loss of less than 0.5%.
LABANet: A Lightweight Asymmetrical Bottleneck and Attention-Based Network for Cloud Detection

Recently, on-board satellite cloud detection has attracted extensive interest due to its ability to reduce invalid data, saving satellite storage and downlink resources. As the demand for accuracy has increased, deep neural networks with encoder-decoder architectures have been widely utilized for cloud detection. Most networks mainly focus on extracting contextual features and fusing more information derived from multilevel features to achieve high accuracy. However, convolutional layers with contextual feature extraction abilities in networks always contain many computations and parameters, making them difficult to deploy in processors with limited computing and memory resources. Additionally, fusing features acquired from different levels easily introduces redundant noise and decreases accuracy. To address these issues, a novel lightweight asymmetrical bottleneck and attention-based network (LABANet) is proposed. In the encoder, a multiscale context-based asymmetrical bottleneck (MCAB) is designed to extract more contextual features in an efficient manner. The numbers of computations and parameters required by the bottleneck can be effectively reduced via asymmetric channel compression and grouping for different pointwise convolutions. Multiscale contextual features can be extracted by dilated depthwise asymmetric convolution. In the decoder, a spatial-channel cascade attention (SCCA) module is proposed to fuse multilevel features with less redundant noise by highlighting spatial and channel correlation information. Experimental results obtained on public datasets demonstrate that the proposed LABANet can exceed 95% overall accuracy (OA) with a computational cost of 1.20 giga-floating-point operations (GFLOPs) and a parameter count of 1.27×10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">6</sup> ; thus, it achieves competitive accuracy with fewer computations and parameters than other baseline cloud detection networks.
Differential private knowledge trading in vehicular federated learning using contract theory

In vehicular differentially private federated learning, the personalized differential privacy mechanism allows vehicles to customize their local privacy budget. However, it leads to uneven utility across datasets. Therefore, model owners must consider the unevenness in utility within the incentive mechanisms to enhance learning performance. Due to limitations in system resources, servers in the Internet of Vehicles should prioritize the selection of high-utility datasets before initiating the learning process. Since existing utility evaluation schemes are costly and inefficient, we propose a knowledge trading framework based on prior utility evaluation and contract theory. Specifically, we first derive a utility evaluation function that captures the functional relationship between utility and privacy elements, such as privacy cost, sensitivity, and learning rounds. These elements are shared as hyperparameters among vehicles and servers for utility evaluation. In the presence of information asymmetry, we utilize the self-revealing property of contract theory to mitigate dishonest behavior. The learning process is modeled as an optimization function that incorporates utility and privacy costs. The complexity is further reduced by utilizing the monotonicity of the utility function. Experimental results based on multiple models and datasets illustrate that our scheme outperforms existing schemes in terms of convergence and accuracy.
Hybrid cross-modal interaction learning for multimodal sentiment analysis

Multimodal sentiment analysis (MSA) predicts the sentiment polarity of an unlabeled utterance that carries multiple modalities, such as text, vision and audio, by analyzing labeled utterances. Existing fusion methods mainly focus on establishing the relationship of characteristics among different modalities to enhance their emotion recognition abilities. However, they always ignore the all-round interaction between different modalities, especially the cross-modal interaction, which is critical to the sentiment decision of multimodal data. To address these issues, we propose a novel hybrid cross-modal interaction learning (HCIL) framework for hybrid learning of intra-modal, inter-modal, interactive-modal and cross-modal interactions, with which the model can fully utilize the sentiment information of multimodalities and enhance the sentiment assistance between modalities. Specifically, we propose two core substructures to learn discriminative multimodal features. One is the comparative learning interaction structure that can track the class dynamics in the intra-modal, reduce the modal gap in the inter-modal and establish emotional communication in the interactive-modal; the other is the cross-modal prediction structure, which can build the sentiment relationship between cross-modal pairs, especially exploring the auxiliary sentiment effect of audio on the vision and text. Furthermore, we adopt a hierarchical feature fusion structure to generate the multimodal feature for the final sentiment prediction. Extensive experiments on three benchmark datasets showed that our HCIL approach can obtain significant performance on the MSA task and that the design of a cross-modal interaction structure can directly promote the improvement of sentiment classification performance.
Towards Unsupervised Referring Expression Comprehension with Visual Semantic Parsing

Referring Expression Comprehension (REC) is a task that involves grounding a specific object in an image based on a given referring query in the form of bounding boxes. Existing fully-supervised or weakly-supervised REC methods rely on either manually annotated regional coordinates or query texts. In this paper, we propose an unsupervised training paradigm for the REC task that does not require any manual annotated data. Specifically, we introduce a Visual-Semantic-Parsing-based Unsupervised Referring Expression Comprehension framework (VUREC), which leverages a Visual Semantic Parser (VSP) as its core module to recognize the rich semantic information in images and construct pseudo-region-query pairs as the training supervision for REC. The VSP comprises a Scene Graph Parser (SGP) and a Visual Concept Detector (VCD) that can detect the locations, categories, attributes of objects, and visual relationships among them in images. Furthermore, we present a Referring Expression Reasoning (RER) model that utilizes a Multi-Modal Cascade Attention Decoder (MCAD) for fine-grained multi-modality fusion and regresses the four coordinates of the referential object directly. The experimental results on three benchmark datasets of Refcoco, Refcoco+ and Refcocog demonstrate the effectiveness of our proposed method.