Reviewerly/exHarmony huggingface dataset:
A Hybrid Convolutional and Transformer Network for Salient Object Detection

We present a novel hybrid architecture that seamlessly merges transformers and convolutional neural networks to enhance the performance of RGB-D salient object detection. Transformer-based models have recently demonstrated their potential in this field, owing to their unique ability to encode long-range information via the self-attention mechanism. This mechanism adeptly mirrors human visual perception by capturing long-distance dependencies and selectively focusing on the most relevant sections of the input image. In contrast, convolutional neural networks, with their robust generalization and trainability, have proven to be invaluable for a wide array of image processing tasks. By fusing the strengths of these two models, our proposed hybrid architecture outperforms the effectiveness of using either transformers or convolutional neural networks in isolation. Our architecture employs an encoder-decoder framework. Within this structure, the hybrid model functions as the feature encoder, while the decoder integrates a convolutional neural network with deep layer aggregation to adeptly merge features of varying resolutions derived from the transformer-based encoder. This strategic design choice exploits the computational modeling prowess of convolutional neural networks in tasks such as saliency prediction, while also benefiting from the long-range dependency modeling offered by the hybrid model. We also use a Siamese architecture with shared parameters in the encoder to concurrently learn salient features from RGB and depth data. By harnessing the complementary strengths of both models, the proposed hybrid architecture has demonstrated superior performance.
Temporal Aggregation with Context Focusing for Few-Shot Video Object Detection

Few-shot video object detection focuses on finding all the objects in a given query video that belong to the same class, given only a few support images of the target object in an unseen class. Unfortunately, due to the object blur or occlusion in video frames, using single-frame object detection directly will greatly limit the accuracy. The issue is significantly worse in few-shot settings due to insufficient support and timedomain information. In this paper, we propose a temporal aggregation with context focusing framework (TACF) for few-shot video object detection, which aims to fully use the information between support images and adjacent video frames. The context focusing module effectively encodes the target object in adjacent frames according to the support images. Afterward, the temporal aggregation module implicitly extracts the most similar ROI features from these adjacent frames to obtain the target proposals. In the end, the matching network determines the category and bounding box by calculating the distance with the support images. Extensive experimental evaluations on FSVOD and FSYTV databases show that our method achieves more competitive results than image-based methods, naive video-based extensions, and the state-of-the-art few-shot video object detection method.
Exploring Global and Local Information for Anomaly Detection with Normal Samples

Anomaly detection aims to detect data that do not conform to regular patterns, and such data is also called outliers. The anomalies to be detected are often tiny in proportion, containing crucial information, and are suitable for application scenes like intrusion detection, fraud detection, fault diagnosis, e-commerce platforms, et al. However, in many realistic scenarios, only the samples following normal behavior are observed, while we can hardly obtain any anomaly information. To address such problem, we propose an anomaly detection method GALDetector which is combined of global and local information based on observed normal samples. The proposed method can be divided into a three-stage method. Firstly, the global similar normal scores and the local sparsity scores of unlabeled samples are computed separately. Secondly, potential anomaly samples are separated from the unlabeled samples corresponding to these two scores and corresponding weights are assigned to the selected samples. Finally, a weighted anomaly detector is trained by loads of samples, then the detector is utilized to identify else anomalies. To evaluate the effectiveness of the proposed method, we conducted experiments on three categories of real-world datasets from diverse domains, and experimental results show that our method achieves better performance when compared with other state-of-the-art methods.
Learning to Communicate Using Action Probabilities for Multi-Agent Cooperation

While communication is an essential tool for cooperation in multi-agent systems (MAS), existing approaches generally assume high-dimensional messages genereated by deep networks, which are uninterpretable for humans and require expensive transmission and complex encoding/decoding networks in agents. Uninterpretability may in turn raise reliability and security issues for the systems. To achieve low-dimensional and interpretable communication, we demonstrate that each agent's action probabilities can be used as messages, inspired by the fact that humans often share likely actions during collaboration. Our proposed method, communication based on action probabilities (CAP), is a simple yet effective communication architecture for coordination and collaboration in MAS, which facilitates our understanding of the agents' learned coordinated and cooperative behaviors. Moreover, because messages can be generated by the actor network, which is usually implemented in deep reinforcement learning agents, we can eliminate the requirement of extra message-generator networks. Our experiments show that CAP can achieve comparable performance to those of the state-of-the-art methods, with quicker convergence, simpler network structures and better interpretability.
A Dynamic Global Semantic Fusion GNN Model For Commonsense Question Answering

Commonsense question answering (CSQA) is a challenging learning task that aims to give correct answers to commonsense questions. CSQA models combining large pretrained language models with knowledge graphs are proposed to perform one-way or two-way information fusion to enhance their commonsense reasoning ability. However, existing CSQA models only fuse local information at the word level, ignoring the global semantic information fusion. Furthermore, current CSQA models often introduce noise nodes when constructing the knowledge subgraph. In addition, existing methods neglect the edge information in message aggregation. To solve these shortcomings, we propose a novel CSQA model named MDEQA. In our model, we design the multi-layer attention fusion module to bidirectionally fuse the word-level local information and global semantic information of question context and knowledge subgraph. Moreover, we design the dynamic graph neural network module with improved GAT and aggregating edge information to form the dynamic subgraphs which alleviate the interference of noise nodes on reasoning and enhance the commonsense reasoning ability of our model. Finally, we evaluated our model on CommonsenseQA and OpenBookQA datasets to compare with other baseline models.
SWDPM: A Social Welfare-Optimized Data Pricing Mechanism

Data trading has been hindered by privacy concerns associated with user-owned data and the infinite reproducibility of data, making it challenging for data owners to retain exclusive rights over their data once it has been disclosed. Traditional data pricing models relied on uniform pricing or subscription-based models. However, with the development of Privacy-Preserving Computing techniques, the market can now protect the privacy and complete transactions using progressively disclosed information, which creates a technical foundation for generating greater social welfare through data usage. In this study, we propose a novel approach to modeling multi-round data trading with progressively disclosed information using a matchmaking-based Markov Decision Process (MDP) and introduce a Social Welfare-optimized Data Pricing Mechanism (SWDPM) to find optimal pricing strategies. To the best of our knowledge, this is the first study to model multi-round data trading with progressively disclosed information. Numerical experiments demonstrate that the SWDPM can increase social welfare 3 times by up to 54 % in trading feasibility, 43 % in trading efficiency, and 25 % in trading fairness by encouraging better matching of demand and price negotiation among traders.
Deploying a Machine Translation Model on a Mobile Device with Improved Latency Constraints

This paper presents a method of deploying a complex machine learning model on a mobile device which has limited computing resources. With the rapid advancement of machine learning and deep learning, Natural Language Processing (NLP) has made considerable progress in recent years. The transformer approach has been widely used in NLP tasks with the ability of parallel processing in recurrent neural networks through the use of a self-attention mechanism. However, the excessive pursuit of accuracy leads to higher complexity in a model, and it becomes difficult to deploy such a model on mobile devices. To solve the problem of high computational cost, this research uses Neural Architecture Search (NAS) with Genetic Algorithm as the search strategy. It trains the supernet for performance evaluation to automatically design efficient architecture for different hardware platforms while considering network compression. We further reduce the size of the model with little impact on accuracy by applying K-means clustering. To show our method's effectiveness, we have deployed a model on a mobile device to perform machine translation. We use BLEU, FLOPs, and Latency as evaluation metrics. BLEU is used as an evaluation of sequence generative tasks while FLOPs and latency are used to observe whether the architecture found by the NAS is suitable for the hardware.
AI-based solutions for predicting sepsis in ICUs

Artificial Intelligence (AI) advances in healthcare underpin timely and informed interventions that can elevate the quality of care for the benefit of the citizen. This can be especially beneficial for Intensive Care Unit (ICU) patients. The goal of this study is to develop an interoperable electronic health record (EHR) system, integrating reproducible AI-based algorithms for ICU services, linked with medical data and healthcare status visualizations, that will allow healthcare professionals to proceed to more informed diagnosis and targeted, personalized treatments. From a data science perspective, AI-based algorithms will first rely on the use of retrospective data via the openly available MIMIC dataset, investigating primarily the early prediction of Sepsis, and secondarily the length of ICU stay and rehospitalization probability. Emphasis will be given to explainability and ethical AI development. Within the context of the “HospAItal” project, real-life pilots will take place at the Nicosia General Hospital (NGH) of Cyprus, assessing the clinical impact of the proposed system while using prospective data to retrain and advance the precision of the developed algorithms.Clinical Relevance: The proposed AI-based system targets timely and informed healthcare interventions for ICU patients at risk of Sepsis, aspiring to increase the quality of care while contributing to more efficient ICU expenditures management.
BASED: Benchmarking, Analysis, and Structural Estimation of Deblurring

This paper discusses the challenges of evaluating deblurring-methods quality and proposes a reduced-reference metric based on machine learning. Traditional quality-assessment metrics such as PSNR and SSIM are common for this task, but not only do they correlate poorly with subjective assessments, they also require ground-truth (GT) frames, which can be difficult to obtain in the case of deblurring. To develop and evaluate our metric, we created a new motion-blur dataset using a beam splitter. The setup captured various motion types using a static camera, as most scenes in existing datasets include blur due to camera motion. We also conducted two large subjective comparisons to aid in metric development. Our resulting metric requires no GT frames, and it correlates well with subjective human perception of blur.
Self-Supervised Optimization of Hand Pose Estimation Using Anatomical Features and Iterative Learning

Manual assembly workers face increasing complexity in their work. Human-centered assistance systems could help, but object recognition as an enabling technology hinders a sophisticated human-centered design of these systems. At the same time, activity recognition based on hand poses suffers from poor pose estimation in complex usage scenarios, such as wearing gloves. This paper presents a self-supervised pipeline for adapting hand pose estimation to specific use cases with minimal human interaction. This enables cheap and robust hand pose-based activity recognition. The pipeline consists of a general machine learning model for hand pose estimation trained on a generalized dataset, spatial and temporal filtering to account for anatomical constraints of the hand, and a retraining step to improve the model. Different parameter combinations are evaluated on a publicly available and annotated dataset. The best parameter and model combination is then applied to unlabeled videos from a manual assembly scenario. The effectiveness of the pipeline is demonstrated by training an activity recognition as a downstream task in the manual assembly scenario.
Low-complexity Transform Network Architecture for JPEG AI Image Codec

Learning-based image coding schemes, exemplified by JPEG AI, have shown potential by greatly exceeding the conventional image compression standards in rate-distortion (RD) performance. However, their widespread applications are hindered by high decoding complexity, particularly from the upsampling and attention modules. Existing works sought to reduce this complexity, but their solutions are not fully effective, leaving considerable complexity unaddressed. In this paper, we present a simplified transform network architecture that employs an optimized attention module, a streamlined upsampling module, and a pared-down activation function to tackle this issue. Simulation results show that the simplified decoder sees its complexity (measured by kMACs/pixel) reduced by 80% (from 833 to 172), while the gain over Versatile Video Coding (VVC) increases slightly (from 27.3% to 27.5%). Partial methods in this paper have been integrated into the JPEG AI Verification Model (VM) software.
Towards Intelligent Training Systems for Customer Service

Customer service is very important in many industrial fields, and the service quality is most essential. However, customer service practitioners are with a high turnover rate, and it usually takes months for a new customer service employee to be an experienced one. If the training of new employees is conducted by other experienced employees, there will be a high resource consumption. Therefore, intelligent training systems for customer service can be designed to replace the manual training. In this paper, we define the task of intelligent training for customer service and propose an architecture of intelligent training systems. Dialogue scripts are prepared offline, and a dialogue simulation module and a service evaluation module are separately intended for the online service training and the service quality evaluation. We evaluate state-of-the-art models with respect to the ability to provide service training, and the experimental results show that our proposed system is effective on this task.
Lightweight Cryptography Implementation for Internet of Things Network on FPGA

With the development of modern communication technology, traditional Internet of Things systems could not provide sufficient support for large data flow, hardware resource, power consumption, and security during transmission. Especially when it comes to the Industrial Internet of Things (IIoT), where the limitations of hardware resource and power consumption are more strict; the requirements of network security and hardware security are much higher. This paper implements a User Datagram Protocol (UDP) communication system, which is encrypted with one of the Lightweight Cryptography, Xoodyak. We aim to design a lightweight encrypted communication platform. Compared with a public key encryption system, our implementation costs much less hardware resource and power consumption, while providing excellent side-channel attack (SCA) protection. Besides, when there is a large burst length of data flow, two asynchronous FIFO can restore those data respectively, so our design can maintain throughput and data integrity in extreme cases. Based on the above properties, our design is relatively ideal in IIoT encryption scenarios.
Tangent Space Sampling of Video Sequence with Locally Structured Unitary Network

This study proposes a novel approach for the analysis of high-dimensional time series data utilizing a linear shift-variant transform. Central to the methodology is the Locally-Structured Unitary Network (LSUN), a self-supervised learning transform proficient in capturing the underlying tangent spaces within high-dimensional datasets. This trait equips LSUN with a distinct advantage over existing techniques such as convolutional dictionary learning, which suffers from decreased interpretability due to the redundant selection of filter kernels. The shift-invariance of convolutional models does not directly correspond to the axes of tangent spaces, a challenge well addressed by LSUN. This study focuses on utilizing LSUN for time series analysis to capture structural variations in sequential frames. To optimize this task, LSUN is utilized with total variation regularization, a method known for its ability to preserve edge information. The effectiveness of the proposed model is demonstrated through a video sequence denoising experiment, illustrating its potential to innovate in areas where the analysis of time-series data is crucial. The findings signal the significant potential for this approach to be extended to targets, such as dynamical system identification, further extending its applicability.
Hybrid Implicit Neural Image Compression with Subpixel Context Model and Iterative Pruner

Recently, image compression based on Implicit Neural Representation (INR) has gained interest. However, the performance of INR-based methods is poor because MLP can not fit high-frequency information well. Recently, a kind of hybrid INR method was proposed, which transmits not only the parameters of MLP but also some external latents to help to overfit at the decoder. Despite its low MACs (Multiply–Accumulate Operations per pixel), the decoding time is still slow due to the serial autoregressive context model for entropy coding. Hence, we propose two methods to improve the hybrid INR method’s decoding speed and performance. First, the Subpixel Context Model (SAM) is aimed to speed up the process of entropy coding. The proposed SAM is latent by latent instead of pixel by pixel. Second, we add an iterative pruner to the pipeline of hybrid INR compression, to further reduce the bit rate. The performance and MACs of our approach are on par with the SOTA of the hybrid INR methods. The decoding speed is about eight times faster on CPU, and three times faster on GPU.
Split-Conv: A Resource-efficient Compression Method for Image Quality Assessment Models

Blind Image Quality Assessment (BIQA) models based on deep neural networks (DNNs) have achieved state-of-the-art performance recently. However, the heavyweight architecture makes them hard to deploy on resource-constrained devices. Filter pruning is one of the most effective ways to compress the DNN model. However, most pruning methods need complete retraining after pruning which is too resource-consuming for IQA models with complex training processes. In this paper, we propose a resource-efficient structural pruning method for IQA models called Split-Conv, where the model only needs to be retrained on small IQA databases. Specifically, we split convolutional layers into sub-convolutional kernels and decorators, which can measure the importance of convolutional channels more precisely, thus reducing the requirement for retraining conditions. The experiments on several IQA models demonstrate the effectiveness of our IQA model pruning approach. For instance, DBCNN, StairIQA, and HyperIQA’s performances can be kept at the baseline level when 50% parameters are pruned with the model being retrained on the authentically distort IQA database containing only 586 images, and the performances of StairIQA and HyperIQA on KonIQ-10k database are still acceptable after 80% of the parameters have been pruned. Besides, due to Split-Conv’s capability to identify the important, useless, and harmful filters, the performances of some BIQA models can even be boosted after model compression with a proper pruning ratio.
depthUNet: A Dehazing Model with Adaptive Depth Attention for Natural Images

Most image dehazing deep learning models target synthetic datasets of hazy images, resulting in not considering features in natural hazy images. Leveraging on depth attention with adaptation, we propose a novel dehazing network called depthUNet, that is focused on natural images. Utilizing the correlation between depth information and haze distribution, our network enhances its generalization performance on natural images. Furthermore, our method improves the PSNR for the non-homogeneous realistic haze dataset NH-HAZE from 20.66 (DeHamer's result) to 20.74, using only 1.6% of the parameters. Similarly, for the outdoor scenes realistic haze dataset O-HAZE, our method enhances the PSNR from 24.36 (MSBDN's result) to 25.77, with just 6% of the parameters. On natural road images with haze from dataset RTTS, our method improved the vehicle detection rate by 10% in terms of the R-squared value. In summary, our method outperforms all state-of-the-art methods while utilizing the least number of parameters.
Learned Progressive Image Compression With Spatial Autoregression

Entropy modeling plays an important role in estimating the rates of latent representations and optimizing the rate-distortion performance for learned image compression. Autoregression modules are demonstrated to eliminate spatial/channel-wise redundancy of latent representations in fixed-rate learned image compression. However, it cannot be efficiently achieved in progressive coding due to the high computational complexity raised by element-wise probability prediction. In this paper, we propose a learned progressive image compression method that enables spatial autoregression for entropy modeling. Specifically, we develop a novel codeword alignment scheme to prevent coding redundancy and achieve efficient autoregression of latent representations in different quality layers. Consequently, conditional probability estimation for the latent prediction can be achieved based on spatial autoregression in a layer-wise manner. We further extend the proposed method by dead-zone quantizers to obtain promoted rate-distortion performance. The proposed method is a successful attempt to enable spatial autoregression in learned progressive coding and further bridge the performance gap with fixed-rate models. Experimental results show that it outperforms traditional methods such as JPEG and BPG, as well as recent fine-grained learned progressive coding models DPICT and PLONQ in terms of rate-distortion performance.