Reviewerly/exHarmony huggingface dataset:
Interpretable Machine Learning for Personalized Medical Recommendations: A LIME-Based Approach

Chronic diseases are increasingly major threats to older persons, seriously affecting their physical health and well-being. Hospitals have accumulated a wealth of health-related data, including patients' test reports, treatment histories, and diagnostic records, to better understand patients' health, safety, and disease progression. Extracting relevant information from this data enables physicians to provide personalized patient-treatment recommendations. While collaborative filtering techniques and classical algorithms such as naive Bayes, logistic regression, and decision trees have had notable success in health-recommendation systems, most current systems primarily inform users of their likely preferences without providing explanations. This paper proposes an approach of deep learning with a local interpretable model-agnostic explanations (LIME)-based interpretable recommendation system to solve this problem. Specifically, we apply the proposed approach to two chronic diseases common in older adults: heart disease and diabetes. After data preprocessing, we use six deep-learning algorithms to form interpretations. In the heart-disease data set, the actual model recommendation of multi-layer perceptron and gradient-boosting algorithm differs from the local model's recommendation of LIME, which can be used as its approximate prediction. From the feature importance of these two algorithms, it can be seen that the CholCheck, GenHith, and HighBP features are the most important for predicting heart disease. In the diabetes data set, the actual model predictions of the multi-layer perceptron and logistic-regression algorithm were little different from the local model's prediction of LIME, which can be used as its approximate recommendation. Moreover, from the feature importance of the two algorithms, it can be seen that the three features of glucose, BMI, and age were the most important for predicting heart disease. Next, LIME is used to determine the importance of each feature that affected the results of the calculated model. Subsequently, we present the contribution coefficients of these features to the final recommendation. By analyzing the impact of different patient characteristics on the recommendations, our proposed system elucidates the underlying reasons behind these recommendations and enhances patient trust. This approach has important implications for medical recommendation systems and encourages informed decision-making in healthcare.
MMS-Net: Multi-level multi-scale feature extraction network for medical image segmentation

Automatic segmentation of medical images can provide a reliable basis for clinical analysis and disease diagnosis. Due to the varying sizes and appearances of different pathological lesions, most convolutional neural networks (CNNs) approaches are unable to fully extract multi-scale features and usually lead to false positive and false negative segmentation. Multi-scale medical image segmentation still faces significant challenges. By designing different levels of skip connection modules (SCMs), local multi-scale context is proposed. We also adopted feature pyramid network (FPN) to connect SCMs to extract global multi-scale features. Local multi-scale context and global multi-scale feature constitute a multi-level multi-scale feature extraction network (MMS-Net). Multi-level multi-scale feature was extracted and accurate spatial information was also retained. Comprehensive experiments (training vs. testing: 8:2) on five public datasets show that the proposed MMS-Net achieved accurate segmentation and outperformed the state-of-the-art algorithms. Besides, when only using 50% and 20% of the training dataset, MMS-Net still obtained competitive segmentation results.
Imbalanced least squares regression with adaptive weight learning

Least squares regression (LSR) has demonstrated promising performance in various classification tasks owing to its effectiveness and efficiency. However, there are some deficiencies that seriously hinder its application in imbalanced data scenarios. The first is that LSR strongly relies on a balanced class distribution. A severely imbalanced class distribution may seriously damage the effectiveness of the algorithm. Second, the utilized binary label matrix in the conventional LSR model may be too strict to learn a discriminative transformation matrix for imbalanced learning. To address the above issues, in this paper, an adaptive weight learning mechanism and label relaxation constraint are proposed and incorporated into the framework of LSR to tackle the imbalanced classification problem. The weight of each sample can be adaptively obtained according to the original distribution information of the imbalanced data, in which the importance of minority class samples can be better reflected with larger weights. A new label relaxation matrix consisting of the original label matrix and auxiliary matrix is constructed to widen the margins between different classes. Further, we provide an iterative algorithm with fast convergence to solve the resulting optimization problem. Extensive experimental results on diverse binary-class and multi-class imbalanced datasets show that the proposed method outperforms many other state-of-the-art imbalanced learning approaches.
AI Ethics on Blockchain: Topic Analysis on Twitter Data for Blockchain Security

Blockchain has empowered computer systems to be more secure using a distributed network. However, the current blockchain design suffers from fairness issues in transaction ordering. Miners are able to reorder transactions to generate profits, the so-called miner extractable value (MEV). Existing research recognizes MEV as a severe security issue and proposes potential solutions, including prominent Flashbots. However, previous studies have mostly analyzed blockchain data, which might not capture the impacts of MEV in a much broader AI society. Thus, in this research, we applied natural language processing (NLP) methods to comprehensively analyze topics in tweets on MEV. We collected more than 20000 tweets with #MEV and #Flashbots hashtags and analyzed their topics. Our results show that the tweets discussed profound topics of ethical concern, including security, equity, emotional sentiments, and the desire for solutions to MEV. We also identify the co-movements of MEV activities on blockchain and social media platforms. Our study contributes to the literature at the interface of blockchain security, MEV solutions, and AI ethics.
A feature-thresholds guided genetic algorithm based on a multi-objective feature scoring method for high-dimensional feature selection

The classical genetic algorithm utilizes random population initialization, an unguided crossover operator, and an unguided mutation operator for feature selection. However, this approach may be too stochastic and result in slow convergence. This paper proposes a hybrid feature selection algorithm named the Feature-Thresholds Guided Genetic Algorithm (FTGGA) to overcome this deficiency. FTGGA first employs ReliefF to filter out redundant features and retains crucial ones. Then, it generates a feature-thresholds set that contains all the feature thresholds. Each feature threshold represents the probability that the corresponding feature will be selected. The feature-thresholds set continuously updates to guide the iteration process of the genetic algorithm, accelerating its convergence. The experimental data demonstrates that FTGGA has a smaller feature subset and better classification accuracy compared to other algorithms.
Enhancing PV panel segmentation in remote sensing images with constraint refinement modules

Incorrect predictions or underestimation of a city's solar potential can result from neglecting common features of photovoltaic (PV) panels from remote sensing images. This paper proposes an improved approach to address the challenge of accurately segmenting PV panels from remote sensing images using deep learning methods. The proposed method incorporates common features of PV panels and a constraint refinement module (CRM) to perform the localization of PV panel regions and shape regularization more accurately. Specifically, the method uses a color loss function based on prior knowledge of color to refine the predicted region with correct color information among confusing objectives, and a shape loss function based on multi-layer shape targets calculation to refine the initial segments and constrain the edge information of predicted regions. Different CRMs are embedded into the four refined initial segment modules, respectively, to improve the detection IoU of PV panels by up to 7.44%. The best CRM-integrated model performs the best IoU of 74.66% when segmenting PV panels. The proposed method has important implications for urban PV panel segmentation at the city level and provides a promising solution for remote sensing image-based PV plate segmentation tasks in challenging scenarios.
A Systematic Literature Review on Cyber Threat Intelligence for Organizational Cybersecurity Resilience

Cybersecurity is a significant concern for businesses worldwide, as cybercriminals target business data and system resources. Cyber threat intelligence (CTI) enhances organizational cybersecurity resilience by obtaining, processing, evaluating, and disseminating information about potential risks and opportunities inside the cyber domain. This research investigates how companies can employ CTI to improve their precautionary measures against security breaches. The study follows a systematic review methodology, including selecting primary studies based on specific criteria and quality valuation of the selected papers. As a result, a comprehensive framework is proposed for implementing CTI in organizations. The proposed framework is comprised of a knowledge base, detection models, and visualization dashboards. The detection model layer consists of behavior-based, signature-based, and anomaly-based detection. In contrast, the knowledge base layer contains information resources on possible threats, vulnerabilities, and dangers to key assets. The visualization dashboard layer provides an overview of key metrics related to cyber threats, such as an organizational risk meter, the number of attacks detected, types of attacks, and their severity level. This relevant systematic study also provides insight for future studies, such as how organizations can tailor their approach to their needs and resources to facilitate more effective collaboration between stakeholders while navigating legal/regulatory constraints related to information sharing.
FlowFormer++: Masked Cost Volume Autoencoding for Pretraining Optical Flow Estimation

FlowFormer [24] introduces a transformer architecture into optical flow estimation and achieves state-of-the-art performance. The core component of FlowFormer is the transformer-based cost-volume encoder. Inspired by the recent success of masked autoencoding (MAE) pretraining in unleashing transformers' capacity of encoding visual representation, we propose Masked Cost Volume Autoencoding (MCVA) to enhance FlowFormer by pretraining the cost-volume encoder with a novel MAE scheme. Firstly, we introduce a block-sharing masking strategy to prevent masked information leakage, as the cost maps of neighboring source pixels are highly correlated. Secondly, we propose a novel pre-text reconstruction task, which encourages the cost-volume encoder to aggregate long-range information and ensures pretraining-finetuning consistency. We also show how to modify the FlowFormer architecture to accommodate masks during pretraining. Pretrained with MCVA, FlowFormer++ ranks 1st among published methods on both Sintel and KITTI-2015 benchmarks. Specifically, FlowFormer++ achieves 1.07 and 1.94 average end-point error (AEPE) on the clean and final pass of Sintel benchmark, leading to 7.76% and 7.18% error reductions from FlowFormer. FlowFormer++ obtains 4.52 F1-all on the KITTI-2015 test set, improving FlowFormer by 0.16.
3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition

Many skeletal action recognition models use GCNs to represent the human body by 3D body joints connected body parts. GCNs aggregate one- or few-hop graph neighbourhoods, and ignore the dependency between not linked body joints. We propose to form hypergraph to model hyperedges between graph nodes (e.g., third- and fourth-order hyper-edges capture three and four nodes) which help capture higher-order motion patterns of groups of body joints. We split action sequences into temporal blocks, Higher-order Transformer (HoT) produces embeddings of each temporal block based on (i) the body joints, (ii) pairwise links of body joints and (iii) higher-order hyper-edges of skeleton body joints. We combine such HoT embeddings of hyper-edges of orders 1,…, r by a novel Multi-order Multi-mode Transformer (3Mformer) with two modules whose order can be exchanged to achieve coupled-mode attention on coupled-mode tokens based on ‘channel-temporal block’, ‘order-channel-body joint’, ‘channel-hyper-edge (any order)’ and ‘channel-only’ pairs. The first module, called Multi-order Pooling (MP), additionally learns weighted aggregation along the hyper-edge mode, whereas the second module, Temporal block Pooling (TP), aggregates along the temporal block <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> For brevity, we write τ temporal blocks per sequence but τ varies. mode. Our end-to-end trainable network yields state-of-the-art results compared to GCN-, transformer- and hypergraph-based counterparts.
Align and Attend: Multimodal Summarization with Dual Contrastive Losses

The goal of multimodal summarization is to extract the most important information from different modalities to form summaries. Unlike unimodal summarization, the multimodal summarization task explicitly leverages cross-modal information to help generate more reliable and high-quality summaries. However, existing methods fail to lever-age the temporal correspondence between different modal-ities and ignore the intrinsic correlation between different samples. To address this issue, we introduce Align and Attend Multimodal Summarization (A2Summ), a unified multimodal transformer-based model which can effectively align and attend the multimodal input. In addition, we propose two novel contrastive losses to model both inter-sample and intra-sample correlations. Extensive experiments on two standard video summarization datasets (TVSum and SumMe) and two multimodal summarization datasets (Daily Mail and CNN) demonstrate the superiority of A2Summ, achieving state-of-the-art performances on all datasets. Moreover, we collected a large-scale multimodal summarization dataset BLiSS, which contains livestream videos and transcribed texts with annotated summaries. Our code and dataset are publicly available at https://boheumd.github.io/A2Summ/.
Unifying Vision, Text, and Layout for Universal Document Processing

We propose Universal Document Processing (UDOP), a foundation Document AI model which unifies text, image, and layout modalities together with varied task formats, including document understanding and generation. UDOP leverages the spatial correlation between textual content and document image to model image, text, and layout modalities with one uniform representation. With a novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain downstream tasks into a prompt-based sequence generation scheme. UDOP is pretrained on both large-scale unlabeled document corpora using innovative self-supervised objectives and diverse labeled data. UDOP also learns to generate document images from text and layout modalities via masked image reconstruction. To the best of our knowledge, this is the first time in the field of document AI that one model simultaneously achieves high-quality neural document editing and content customization. Our method sets the state-of-the-art on 8 Document AI tasks, e.g., document understanding and QA, across diverse data domains like finance reports, academic papers, and web-sites. UDOP ranks first on the leaderboard of the Document Understanding Benchmark. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code and models: https://github.com/microsoft/i-Code/tree/main/i-Code-Doc
Deep Deterministic Uncertainty: A New Simple Baseline

Reliable uncertainty from deterministic single-forward pass models is sought after because conventional methods of uncertainty quantification are computationally expensive. We take two complex single-forward-pass uncertainty approaches, DUQ and SNGP, and examine whether they mainly rely on a well-regularized feature space. Crucially, without using their more complex methods for estimating uncertainty, we find that a single softmax neural net with such a regularized feature-space, achieved via residual connections and spectral normalization, outperforms DUQ and SNGP's epistemic uncertainty predictions using simple Gaussian Discriminant Analysis post-training as a separate feature-space density estimator-without fine-tuning on OoD data, feature ensembling, or input pre-procressing. Our conceptually simple Deep Deterministic Uncertainty (DDU) baseline can also be used to disentangle aleatoric and epistemic uncertainty and performs as well as Deep Ensembles, the state-of-the art for uncertainty prediction, on several OoD bench-marks (CIFAR-10/100 vs SVHN/Tiny-ImageNet, ImageNet vs ImageNet-O), active learning settings across different model architectures, as well as in large scale vision tasks like semantic segmentation, while being computationally cheaper.
Global and Local Mixture Consistency Cumulative Learning for Long-tailed Visual Recognitions

In this paper, our goal is to design a simple learning paradigm for long-tail visual recognition, which not only improves the robustness of the feature extractor but also alleviates the bias of the classifier towards head classes while reducing the training skills and overhead. We propose an efficient one-stage training strategy for long-tailed visual recognition called Global and Local Mixture Consistency cumulative learning (GLMC). Our core ideas are twofold: (1) a global and local mixture consistency loss improves the robustness of the feature extractor. Specifically, we generate two augmented batches by the global MixUp and local CutMix from the same batch data, respectively, and then use cosine similarity to minimize the difference. (2) A cumulative head-tail soft label reweighted loss mitigates the head class bias problem. We use empirical class frequencies to reweight the mixed label of the head-tail class for long-tailed data and then balance the conventional loss and the rebalanced loss with a coefficient accumulated by epochs. Our approach achieves state-of-the-art accuracy on CIFAR10-LT, CIFAR100-LT, and ImageNet-LT datasets. Additional experiments on balanced ImageNet and CIFAR demonstrate that GLMC can significantly improve the gen-eralization of backbones. Code is made publicly available at https://github.com/ynu-yangpeng/GLMC.
FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation

Recent attention in instance segmentation has focused on query-based models. Despite being non-maximum suppression (NMS)-free and end-to-end, the superiority of these models on high-accuracy real-time benchmarks has not been well demonstrated. In this paper, we show the strong potential of query-based models on efficient instance segmentation algorithm designs. We present FastInst, a simple, effective query-based framework for real-time instance segmentation. FastInst can execute at a real-time speed (i.e., 32.5 FPS) while yielding an AP of more than 40 (i.e., 40.5 AP) on COCO test-dev without bells and whistles. Specifically, FastInst follows the meta-architecture of recently introduced Mask2Former. Its key designs include instance activation-guided queries, dual-path update strategy, and ground truth mask-guided learning, which enable us to use lighter pixel decoders, fewer Transformer decoder layers, while achieving better performance. The experiments show that FastInst outperforms most state-of-the-art real-time counterparts, including strong fully convolutional baselines, in both speed and accuracy. Code can be found at https://github.com/junjiehe96/FastInst.
Detecting Human-Object Contact in Images

Humans constantly contact objects to move and perform tasks. Thus, detecting human-object contact is important for building human-centered artificial intelligence. However, there exists no robust method to detect contact between the body and the scene from an image, and there exists no dataset to learn such a detector. We fill this gap with HOT (“Human-Object conTact”), a new dataset of human-object contacts in images. To build HOT, we use two data sources: (1) We use the PROX dataset of 3D human meshes moving in 3D scenes, and automatically annotate 2D image areas for contact via 3D mesh proximity and projection. (2) We use the V-COCO, HAKE and Watch-n-Patch datasets, and ask trained annotators to draw polygons around the 2D image areas where contact takes place. We also annotate the involved body part of the human body. We use our HOT dataset to train a new contact detector, which takes a single color image as input, and outputs 2D contact heatmaps as well as the body-part labels that are in contact. This is a new and challenging task, that extends current foot-ground or hand-object contact detectors to the full generality of the whole body. The detector uses a part-attention branch to guide contact estimation through the context of the surrounding body parts and scene. We evaluate our detector extensively, and quantitative results show that our model outperforms baselines, and that all components contribute to better performance. Results on images from an online repository show reasonable detections and generalizability. Our HOT data and model are available for research at https://hot.is.tue.mpg.de.
Extracting Motion and Appearance via Inter-Frame Attention for Efficient Video Frame Interpolation

Effectively extracting inter-frame motion and appearance information is important for video frame interpolation (VFI). Previous works either extract both types of information in a mixed way or devise separate modules for each type of information, which lead to representation ambiguity and low efficiency. In this paper, we propose a new module to explicitly extract motion and appearance information via a unified operation. Specifically, we rethink the information process in inter-frame attention and reuse its attention map for both appearance feature enhancement and motion information extraction. Furthermore, for efficient VFI, our proposed module could be seamlessly integrated into a hybrid CNN and Transformer architecture. This hybrid pipeline can alleviate the computational complexity of inter-frame attention as well as preserve detailed low-level structure information. Experimental results demonstrate that, for both fixed- and arbitrary-timestep interpolation, our method achieves state-of-the-art performance on various datasets. Meanwhile, our approach enjoys a lighter computation overhead over models with close performance. The source code and models are available at https://github.com/MCG-NJU/EMA-VFI.