Reviewerly/exHarmony huggingface dataset:
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey

Abstract With the urgent demand for generalized deep models, many pre-trained big models are proposed, such as bidirectional encoder representations (BERT), vision transformer (ViT), generative pre-trained transformers (GPT), etc. Inspired by the success of these models in single domains (like computer vision and natural language processing), the multi-modal pre-trained big models have also drawn more and more attention in recent years. In this work, we give a comprehensive survey of these models and hope this paper could provide new insights and helps fresh researchers to track the most cutting-edge works. Specifically, we firstly introduce the background of multi-modal pre-training by reviewing the conventional deep learning, pre-training works in natural language process, computer vision, and speech. Then, we introduce the task definition, key challenges, and advantages of multi-modal pre-training models (MM-PTMs), and discuss the MM-PTMs with a focus on data, objectives, network architectures, and knowledge enhanced pre-training. After that, we introduce the downstream tasks used for the validation of large-scale MM-PTMs, including generative, classification, and regression tasks. We also give visualization and analysis of the model parameters and results on representative downstream tasks. Finally, we point out possible research directions for this topic that may benefit future works. In addition, we maintain a continuously updated paper list for large-scale pre-trained multi-modal big models: https://github.com/wangxiao5791509/MultiModal_BigModels_Survey .
SRL-ACO: A text augmentation framework based on semantic role labeling and ant colony optimization

The process of creating high-quality labeled data is crucial for training machine-learning models, but it can be a time-consuming and labor-intensive process. Moreover, manual annotation by human annotators can lead to varying degrees of competency, training, and experience, which can result in inconsistent labeling and arbitrary standards. To address these challenges, researchers have been exploring automated methods for enhancing training and testing datasets. This paper proposes SRL-ACO, a novel text augmentation framework that leverages Semantic Role Labeling (SRL) and Ant Colony Optimization (ACO) techniques to generate additional training data for natural language processing (NLP) models. The framework uses SRL to identify the semantic roles of words in a sentence and ACO to generate new sentences that preserve these roles. SRL-ACO can enhance the accuracy of NLP models by generating additional data without requiring manual data annotation. The paper presents experimental results demonstrating the effectiveness of SRL-ACO on seven text classification datasets for sentiment analysis, toxic text detection and sarcasm identification. The results show that SRL-ACO improves the performance of a classifier on different NLP tasks. These results demonstrate that SRL-ACO has the potential to enhance the quality and quantity of training data for various NLP tasks.
Optimized hybrid YOLOu‐Quasi‐ProtoPNet for insulators classification

Abstract To ensure the electrical power supply, inspections are frequently performed in the power grid. Nowadays, several inspections are conducted considering the use of aerial images since the grids might be in places that are difficult to access. The classification of the insulators' conditions recorded in inspections through computer vision is challenging, as object identification methods can have low performance because they are typically pre‐trained for a generalized task. Here, a hybrid method called YOLOu‐Quasi‐ProtoPNet is proposed for the detection and classification of failed insulators. This model is trained from scratch, using a personalized ultra‐large version of YOLOv5 for insulator detection and the optimized Quasi‐ProtoPNet model for classification. For the optimization of the Quasi‐ProtoPNet structure, the backbones VGG‐16, VGG‐19, ResNet‐34, ResNet‐152, DenseNet‐121, and DenseNet‐161 are evaluated. The F1‐score of 0.95165 was achieved using the proposed approach (based on DenseNet‐161) which outperforms models of the same class such as the Semi‐ProtoPNet, Ps‐ProtoPNet, Gen‐ProtoPNet, NP‐ProtoPNet, and the standard ProtoPNet for the classification task.
Role of financial inclusion and ICT for sustainable economic development in developing countries

Sustainable economic development and its close link with financial inclusion have gained so much attention in the empirical literature in the past. However, with the advent of information and communication technology and its association with financial platforms, a system has emerged that offers new opportunities to fill the wealth gaps in the developing world. Empirical investigations are rare in the context of the developing region, and this study strives to fill this gap. We aim to investigate this link for developing countries based on innovation diffusion theory. For this purpose, we select an extended sample of 110 developing countries from 2004 to 2020. Through principal component analysis, we formulate composite financial inclusion and ICT indicators based on broader attributes. We employed Driscoll–Kraay standard errors (D-K), Generalized linear model (GLM), feasible general least squares (FGLS), and difference GMM to obtain compressive and consistent regression coefficients without the need to address temporal dependence and heteroscedasticity. The robust panel estimation illustrates a significantly positive effect of financial inclusion, information communication and technology and sustainable economic development. Our results hold for subsamples based on different income levels of countries. The policymakers are recommended to develop policies for an inclusive financial system and access to ICT on an equitable basis to ensure sustainable economic growth.
Deformable convolution and coordinate attention for fast cattle detection

Cattle detection is an important task in precision livestock farming, but it remains challenging due to the varying appearance and poses of cattle in different scenarios. In this paper, we propose a novel approach for fast cattle detection using deformable convolution and coordinate attention within YOLOv8, a SOTA object detection model. Our proposed method enhances the YOLOv8 architecture by introducing deformable convolution to capture more fine-grained spatial information and coordinate attention to emphasize important features in the detection process. We evaluate our method on a cattle dataset collected in a cattle farm and achieve superior performance compared to the baseline YOLOv8 and several SOTA object detection models. Specifically, our approach achieves a mean average precision (mAP) of 72.9% at 62.5 frames per second (FPS), which demonstrates its effectiveness and efficiency for fast cattle detection. By deploying our method on the farm’s monitoring computer, our proposed approach has the potential to facilitate the development of automated cattle monitoring systems for improving animal welfare and farm management.
A comprehensive review of deep neural networks for medical image processing: Recent developments and future opportunities

Artificial Intelligence (AI) solutions have been widely used in healthcare, and recent developments in deep neural networks have contributed to significant advances in medical image processing. Much ongoing research is aimed at helping medical practitioners by providing automated systems to analyze images and diagnose acute diseases, such as brain tumors, bone cancer, breast cancer, bone fracture, and many others. This comprehensive review delivers an overview of recent advances in medical imaging using deep neural networks. In addition to the comprehensive literature review, a summary of openly available data sources and future research directions are outlined.
Electrical programmable multilevel nonvolatile photonic random-access memory

Photonic Random-Access Memories (P-RAM) are an essential component for the on-chip non-von Neumann photonic computing by eliminating optoelectronic conversion losses in data links. Emerging Phase-Change Materials (PCMs) have been showed multilevel memory capability, but demonstrations still yield relatively high optical loss and require cumbersome WRITE-ERASE approaches increasing power consumption and system package challenges. Here we demonstrate a multistate electrically programmed low-loss nonvolatile photonic memory based on a broadband transparent phase-change material (Ge2Sb2Se5, GSSe) with ultralow absorption in the amorphous state. A zero-static-power and electrically programmed multi-bit P-RAM is demonstrated on a silicon-on-insulator platform, featuring efficient amplitude modulation up to 0.2 dB/μm and an ultralow insertion loss of total 0.12 dB for a 4-bit memory showing a 100× improved signal to loss ratio compared to other phase-change-materials based photonic memories. We further optimize the positioning of dual microheaters validating performance tradeoffs. Experimentally we demonstrate a half-a-million cyclability test showcasing the robust approach of this material and device. Low-loss photonic retention-of-state adds a key feature for photonic functional and programmable circuits impacting many applications including neural networks, LiDAR, and sensors for example.
NTIRE 2023 Challenge on HR Depth from Images of Specular and Transparent Surfaces

This paper reports about the NTIRE 2023 challenge on HR Depth From images of Specular and Transparent surfaces, held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2023. This challenge is held to boost the research on depth estimation, mainly to deal with two of the open issues in the field: high-resolution images and non-Lambertian surfaces characterizing specular and transparent materials. The challenge is divided into two tracks: a stereo track focusing on disparity estimation from rectified pairs and a mono track dealing with single-image depth estimation. The challenge attracted about 100 registered participants for the two tracks. In the final testing stage, 5 participating teams submitted their models and fact sheets, 2 and 3 for the Stereo and Mono tracks, respectively.
NTIRE 2023 Challenge on Image Denoising: Methods and Results

This paper reviews the NTIRE 2023 challenge on image denoising (σ = 50) with a focus on the proposed solutions and results. The aim is to obtain a network design capable to produce high-quality results with the best performance measured by PSNR for image denoising. Independent additive white Gaussian noise (AWGN) is assumed and the noise level is 50. The challenge had 225 registered participants, and 16 teams made valid submissions. They gauge the state-of-the-art for image denoising.
Blockchain adoption in sustainable supply chains for Industry 5.0: A multistakeholder perspective

Industry 5.0 has introduced a novel interpretation of sustainable supply chains (SSCs) that emphasizes the importance of building a transparent and trustworthy network that can be continuously monitored and controlled through stakeholder collaboration as well as the use of advanced, intelligent machinery. The ultimate goal of SSCs is to meet specific economic, social, and environmental standards. The implementation of blockchain technology can significantly improve the reliability, efficiency, and security of the information exchanged among stakeholders in SSCs. However, these stakeholders inevitably possess varying informational advantages and exhibit divergent perspectives regarding the adoption of blockchain technology. This paper thus aims to examine the impediments to the adoption of blockchain technology in the context of SSCs with the goal of promoting blockchain adoption. To achieve this objective, this study analyzes the barriers to blockchain adoption from the perspectives of various stakeholders in SSCs and constructs a barrier severity assessment model that utilizes group decision-making methods to integrate all stakeholders' attitudes. This study employs the PEEST (political, economic, environmental, social, and technological) framework to identify 27 barriers to the adoption of blockchain technology. Subsequently, an expertise-based group decision-making approach is used to quantify the prominence of various barriers according to various types of stakeholders. The results indicate that the five most intense barriers are storage constraints, insufficient economic incentives, high integration costs, a lack of functional appeal, and ambiguity regarding data disclosure and public data management regulations. This research makes novel theoretical and practical contributions, as it takes an empirical and all-encompassing approach to identifying obstacles to the adoption of blockchain technology and provides valuable insights for policymakers and practitioners to reference in overcoming these obstacles.
Slimmable Dataset Condensation

Dataset distillation, also known as dataset condensation, aims to compress a large dataset into a compact synthetic one. Existing methods perform dataset condensation by assuming a fixed storage or transmission budget. When the budget changes, however, they have to repeat the synthesizing process with access to original datasets, which is highly cumbersome if not infeasible at all. In this paper, we explore the problem of slimmable dataset condensation, to extract a smaller synthetic dataset given only previous condensation results. We first study the limitations of existing dataset condensation algorithms on such a successive compression setting and identify two key factors: (1) the inconsistency of neural networks over different compression times and (2) the underdetermined solution space for synthetic data. Accordingly, we propose a novel training objective for slimmable dataset condensation to explicitly account for both factors. Moreover, synthetic datasets in our method adopt a significance-aware parameterization. Theoretical derivation indicates that an upper-bounded error can be achieved by discarding the minor components without training. Alternatively, if training is allowed, this strategy can serve as a strong initialization that enables a fast convergence. Extensive comparisons and ablations demonstrate the superiority of the proposed solution over existing methods on multiple benchmarks.
K-Planes: Explicit Radiance Fields in Space, Time, and Appearance

We introduce k-planes, a white-box model for radiance fields in arbitrary dimensions. Our model uses planes to represent a d-dimensional scene, providing a seamless way to go from static (d = 3) to dynamic (d= 4) scenes. This planar factorization makes adding dimension-specific priors easy, e.g. temporal smoothness and multi-resolution spatial structure, and induces a natural decomposition of static and dynamic components of a scene. We use a linear feature decoder with a learned color basis that yields similar performance as a nonlinear black-box MLP decoder. Across a range of synthetic and real, static and dynamic, fixed and varying appearance scenes, k-planes yields competitive and often state-of-the-art recon- struction fidelity with low memory usage, achieving 1000x compression over a full 4D grid, and fast optimization with a pure PyTorch implementation. For video results and code, please see sarafridov.github.io/K-Planes.
TFL-DT: A Trust Evaluation Scheme for Federated Learning in Digital Twin for Mobile Networks

Due to the distributed collaboration and privacy protection features, federated learning is a promising technology to perform the model training in virtual twins of Digital Twin for Mobile Networks (DTMN). In order to enhance the reliability of the model, it is always expected that the users involved in federated learning have trustworthy behaviors. Yet, available trust evaluation schemes for federated learning have the problems of considering simplex evaluation factor and using coarse-grained trust calculation method. In this paper, we propose a trust evaluation scheme for federated learning in DTMN, which takes direct trust evidence and recommended trust information into account. A user behavior model is designed based on multiple attributes to depict users’ behavior in a fine-grained manner. Furthermore, the trust calculation methods for local trust value and recommended trust value of a user are proposed using the data of user behavior model as trust evidence. Several experiments were conducted to verify the effectiveness of the proposed scheme. The results show that the proposed method is able to evaluate the trust levels of users with different behavior patterns accurately. Moreover, it performs better in resisting attacks from users that alternately execute good and bad behaviors compared with state-of-the-art scheme.
Deep OC-Sort: Multi-Pedestrian Tracking by Adaptive Re-Identification

Motion-based association for Multi-Object Tracking (MOT) has recently re-achieved prominence with the rise of powerful object detectors. Despite this, little work has been done to incorporate appearance cues beyond simple heuristic models that lack robustness to feature degradation. In this paper, we propose a novel way to leverage objects’ appearances to adaptively integrate appearance matching into existing high-performance motion-based methods. Building upon the pure motion-based method OC-SORT, we achieve 1st place on MOT20 and 2nd place on MOT17 with 63.9 and 64.9 HOTA, respectively. We also achieve 61.3 HOTA on the challenging DanceTrack benchmark as a new state-of-the-art even compared to more heavily-designed methods. The code and models are available at https://github.com/GerardMaggiolino/Deep-OC-SORT.
FastReID: A Pytorch Toolbox for General Instance Re-identification

General Instance Re-identification is a very important task in computer vision, which can be widely used in many practical applications, such as person/vehicle re-identification, face recognition, wildlife protection, commodity tracing, snapshots, and so on. To meet the increasing application demand for general instance re-identification, we present FastReID as a widely used software system. In FastReID, the highly modular and extensible design makes it easy for the researcher to achieve new research ideas. Friendly manageable system configuration and engineering deployment functions allow practitioners to quickly deploy models into productions. We have implemented some state-of-the-art projects, including person re-id, partial re-id, cross-domain re-id, and vehicle re-id. Moreover, we plan to release these pre-trained models on multiple benchmark datasets. FastReID is by far the most general and high-performance toolbox that supports single and multiple GPU servers, it can reproduce our project results very easily. The source codes and models have been released at https://github.com/JDAI-CV/fast-reid.
Width-Based Planning for General Video-Game Playing

IW(1) is a simple search algorithm that assumes that states can be characterized in terms of a set of boolean features or atoms. IW(1) consists of a standard breadth-first search with one variation: a newly generated state is pruned if it does not make a new atom true. Thus, while a breadth-first search runs in time that is exponential in the number of atoms, IW(1) runs in linear time. Variations of the algorithm have been shown to yield state-of-the-art results in classical planning and more recently in the Atari video games. In this paper, we use the algorithm for selecting actions in the games of the general video-game AI competition (GVG-AI) which, unlike classical planning problems and the Atari games, are stochastic. We evaluate a variation of the algorithm over 30 games under different time windows using the number of wins as the performance measure. We find that IW(1) does better than the sample MCTS and OLMCTS controllers for all time windows with the performance gap growing with the window size. The exception are the puzzle-like games where all the algorithms do poorly. For such problems, we show that much better results can be obtained with the IW(2) algorithm, which is like IW(1), except that states are pruned in the breadth-first search when they fail to make true a new pair of atoms.
Eyetracking-based assessment of affect-related decay of human performance in visual tasks

Computerized recognition of human emotions enriches the man–machine interaction with a new personal and behavioral aspect. Firstly, because the influence of emotions on the human performance can be objectively assessed and predicted using standardized stimuli, secondly, because an emotion-sensitive computer system can be programmed to react adequately. This paper presents an investigation of emotional influence on the human scanpath. The proposed method assumes that the emotional state of the observer can distract his or her visual attention and can be reliably expressed in parameters of eye movements. We performed visual task experiments in order to record scanpaths from volunteers under auditory stress of controlled intensity. We used a calibrated set of stimuli and controlled the response of volunteers’ central nervous system expressed by the heart rate. For each efficiently stimulated participant, we related the scanpath parameters to the accuracy of solving a given task and to participant’s comment. As a principal result, we obtained a significant change of (1) saccade frequency in 90% and (2) maximum velocity in fixation phase in 80% of participants under stress. These results prove the influence of emotions on visual acuity, feasibility of eyetracking-based assessment of emotional state and motivate further investigations.
High-Order Moment Filtering for Markov Jump Systems in Finite Frequency Domain

Considering that state distribution obeys a generalized Gaussian distribution, it cannot be accurately estimated using only the mean error criterion or mean square error criterion. Therefore, this brief addresses the high-order moment filter design problem for Markov jump systems in the finite frequency domain. An estimation method is applied by transforming the original stochastic system into a high-order component form via the cumulant generating function. Then, the generalized Kalman-Yakubovic-Popov lemma can be used in filter design, by considering finite frequency characteristics. Sufficient conditions for the existence of a high-order moment filter with the required finite frequency performance are established. Finally, a simulation example is presented to verify the effectiveness of the proposed method.