OpenRaiser/Intern-Atlas huggingface dataset:
URM4DMU: an user represention model for darknet markets users
Venue: arXiv cs.LG 2023
Year: 2023

Darknet markets provide a large platform for trading illicit goods and services due to their anonymity. Learning an invariant representation of each user based on their posts on different markets makes it easy to aggregate user information across different platforms, which helps identify anonymous users. Traditional user representation methods mainly rely on modeling the text information of posts and cannot capture the temporal content and the forum interaction of posts. While recent works mainly use CNN to model the text information of posts, failing to effectively model posts whose length changes frequently in an episode. To address the above problems, we propose a model named URM4DMU(User Representation Model for Darknet Markets Users) which mainly improves the post representation by augmenting convolutional operators and self-attention with an adaptive gate mechanism. It performs much better when combined with the temporal content and the forum interaction of posts. We demonstrate the effectiveness of URM4DMU on four darknet markets. The average improvements on MRR value and Recall@10 are 22.5% and 25.5% over the state-of-the-art method respectively.
FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering
Venue: arXiv cs.CL 2023
Year: 2023

The widely used Fact-based Visual Question Answering (FVQA) dataset contains visually-grounded questions that require information retrieval using common sense knowledge graphs to answer. It has been observed that the original dataset is highly imbalanced and concentrated on a small portion of its associated knowledge graph. We introduce FVQA 2.0 which contains adversarial variants of test questions to address this imbalance. We show that systems trained with the original FVQA train sets can be vulnerable to adversarial samples and we demonstrate an augmentation scheme to reduce this vulnerability without human annotations.
Evaluation of Convolution Primitives for Embedded Neural Networks on 32-bit Microcontrollers
Venue: arXiv cs.LG 2023
Year: 2023

Deploying neural networks on constrained hardware platforms such as 32-bit microcontrollers is a challenging task because of the large memory, computing and energy requirements of their inference process. To tackle these issues, several convolution primitives have been proposed to make the standard convolution more computationally efficient. However, few of these primitives are really implemented for 32-bit microcontrollers. In this work, we collect different state-of-the-art convolutional primitives and propose an implementation for ARM Cortex-M processor family with an open source deployment platform (NNoM). Then, we carry out experimental characterization tests on these implementations. Our benchmark reveals a linear relationship between theoretical MACs and energy consumption. Thus showing the advantages of using computationally efficient primitives like shift convolution. We discuss about the significant reduction in latency and energy consumption due to the use of SIMD instructions and highlight the importance of data reuse in those performance gains. For reproducibility purpose and further experiments, codes and experiments are publicly available.
Modeling and Mitigating Online Misinformation: a Suggested Blockchain Approach
Venue: arXiv cs.SI 2023
Year: 2023

Misinformation propagation in online social networks has become an increasingly challenging problem. Although many studies exist to solve the problem computationally, a permanent and robust solution is yet to be discovered. In this study, we propose and demonstrate the effectiveness of a blockchain-machine learning hybrid approach for addressing the issue of misinformation in a crowdsourced environment. First, we motivate the use of blockchain for this problem by finding the crucial parts contributing to the dissemination of misinformation and how blockchain can be useful, respectively. Second, we propose a method that combines the wisdom of the crowd with a behavioral classifier to classify the news stories in terms of their truthfulness while reducing the effects of the actions performed by malicious users. We conduct experiments and simulations under different scenarios and attacks to assess the performance of this approach. Finally, we provide a case study involving a comparison with an existing approach using Twitter Birdwatch data. Our results suggest that this solution holds promise and warrants further investigation.
MXM-CLR: A Unified Framework for Contrastive Learning of Multifold Cross-Modal Representations
Venue: arXiv cs.CV 2023
Year: 2023

Multifold observations are common for different data modalities, e.g., a 3D shape can be represented by multi-view images and an image can be described with different captions. Existing cross-modal contrastive representation learning (XM-CLR) methods such as CLIP are not fully suitable for multifold data as they only consider one positive pair and treat other pairs as negative when computing the contrastive loss. In this paper, we propose MXM-CLR, a unified framework for contrastive learning of multifold cross-modal representations. MXM-CLR explicitly models and learns the relationships between multifold observations of instances from different modalities for more comprehensive representation learning. The key of MXM-CLR is a novel multifold-aware hybrid loss which considers multiple positive observations when computing the hard and soft relationships for the cross-modal data pairs. We conduct quantitative and qualitative comparisons with SOTA baselines for cross-modal retrieval tasks on the Text2Shape and Flickr30K datasets. We also perform extensive evaluations on the adaptability and generalizability of MXM-CLR, as well as ablation studies on the loss design and effects of batch sizes. The results show the superiority of MXM-CLR in learning better representations for the multifold data. The code is available at https://github.com/JLU-ICL/MXM-CLR.
PanGu-Σ: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing
Venue: arXiv cs.CL 2023
Year: 2023

The scaling of large language models has greatly improved natural language understanding, generation, and reasoning. In this work, we develop a system that trained a trillion-parameter language model on a cluster of Ascend 910 AI processors and MindSpore framework, and present the language model with 1.085T parameters named PanGu-Σ. With parameter inherent from PanGu-α, we extend the dense Transformer model to sparse one with Random Routed Experts (RRE), and efficiently train the model over 329B tokens by using Expert Computation and Storage Separation(ECSS). This resulted in a 6.3x increase in training throughput through heterogeneous computing. Our experimental findings show that PanGu-Σ provides state-of-the-art performance in zero-shot learning of various Chinese NLP downstream tasks. Moreover, it demonstrates strong abilities when fine-tuned in application data of open-domain dialogue, question answering, machine translation and code generation.
Revisiting Realistic Test-Time Training: Sequential Inference and Adaptation by Anchored Clustering Regularized Self-Training
Venue: arXiv cs.LG 2023
Year: 2023

Deploying models on target domain data subject to distribution shift requires adaptation. Test-time training (TTT) emerges as a solution to this adaptation under a realistic scenario where access to full source domain data is not available, and instant inference on the target domain is required. Despite many efforts into TTT, there is a confusion over the experimental settings, thus leading to unfair comparisons. In this work, we first revisit TTT assumptions and categorize TTT protocols by two key factors. Among the multiple protocols, we adopt a realistic sequential test-time training (sTTT) protocol, under which we develop a test-time anchored clustering (TTAC) approach to enable stronger test-time feature learning. TTAC discovers clusters in both source and target domains and matches the target clusters to the source ones to improve adaptation. When source domain information is strictly absent (i.e. source-free) we further develop an efficient method to infer source domain distributions for anchored clustering. Finally, self-training~(ST) has demonstrated great success in learning from unlabeled data and we empirically figure out that applying ST alone to TTT is prone to confirmation bias. Therefore, a more effective TTT approach is introduced by regularizing self-training with anchored clustering, and the improved model is referred to as TTAC++. We demonstrate that, under all TTT protocols, TTAC++ consistently outperforms the state-of-the-art methods on five TTT datasets, including corrupted target domain, selected hard samples, synthetic-to-real adaptation and adversarially attacked target domain. We hope this work will provide a fair benchmarking of TTT methods, and future research should be compared within respective protocols.
An Improved Exact Algorithm for Knot-Free Vertex Deletion
Venue: arXiv cs.DS 2023
Year: 2023

A knot $K$ in a directed graph $D$ is a strongly connected component of size at least two such that there is no arc $(u,v)$ with $u \in V(K)$ and $v\notin V(K)$. Given a directed graph $D=(V,E)$, we study Knot-Free Vertex Deletion (KFVD), where the goal is to remove the minimum number of vertices such that the resulting graph contains no knots. This problem naturally emerges from its application in deadlock resolution since knots are deadlocks in the OR-model of distributed computation. The fastest known exact algorithm in literature for KFVD runs in time $\mathcal{O}^\star(1.576^n)$. In this paper, we present an improved exact algorithm running in time $\mathcal{O}^\star(1.4549^n)$, where $n$ is the number of vertices in $D$. We also prove that the number of inclusion wise minimal knot-free vertex deletion sets is $\mathcal{O}^\star(1.4549^n)$ and construct a family of graphs with $Ω(1.4422^n)$ minimal knot-free vertex deletion sets
Timely Status Update in Relay-Assisted Cooperative Communications
Venue: arXiv cs.IT 2023
Year: 2023

We investigate the age of information (AoI) of a relay-assisted cooperative communication system, where a source node sends status update packets to the destination node as timely as possible with the aid of a relay node. For time-slotted systems without relaying, prior works have shown that the source should generate and send a new packet to the destination every time slot to minimize the average AoI, regardless of whether the destination has successfully decoded the packet in the previous slot. However, when a dedicated relay is involved, whether the relay can improve the AoI performance requires an in-depth study. In particular, the packet generation and transmission strategy of the source should be carefully designed to cooperate with the relay. Depending on whether the source and the relay are allowed to transmit simultaneously, two relay-assisted schemes are investigated: time division multiple access (TDMA) and non-orthogonal multiple access (NOMA) schemes. A key challenge in deriving their theoretical average AoI is that the destination has different probabilities of successfully receiving an update packet in different time slots. We model each scheme using a Markov chain to derive the corresponding closed-form average AoI. Interestingly, our theoretical analysis indicates that the relay-assisted schemes can only outperform the non-relay scheme in average AoI when the signal-to-noise ratio of the source-destination link is below -2dB. Furthermore, comparing the merits of relay-assisted schemes, simulation results show that the TDMA scheme has a lower energy consumption, while the NOMA counterpart typically achieves a lower average AoI.
HMC: Hierarchical Mesh Coarsening for Skeleton-free Motion Retargeting
Venue: arXiv cs.CV 2023
Year: 2023

We present a simple yet effective method for skeleton-free motion retargeting. Previous methods transfer motion between high-resolution meshes, failing to preserve the inherent local-part motions in the mesh. Addressing this issue, our proposed method learns the correspondence in a coarse-to-fine fashion by integrating the retargeting process with a mesh-coarsening pipeline. First, we propose a mesh-coarsening module that coarsens the mesh representations for better motion transfer. This module improves the ability to handle small-part motion and preserves the local motion interdependence between neighboring mesh vertices. Furthermore, we leverage a hierarchical refinement procedure to complement missing mesh details by gradually improving the low-resolution mesh output with a higher-resolution one. We evaluate our method on several well-known 3D character datasets, and it yields an average improvement of 25% on point-wise mesh euclidean distance (PMD) against the start-of-art method. Moreover, our qualitative results show that our method is significantly helpful in preserving the moving consistency of different body parts on the target character due to disentangling body-part structures and mesh details in a hierarchical way.
Controllable Ancient Chinese Lyrics Generation Based on Phrase Prototype Retrieving
Venue: arXiv cs.CL 2023
Year: 2023

Generating lyrics and poems is one of the essential downstream tasks in the Natural Language Processing (NLP) field. Current methods have performed well in some lyrics generation scenarios but need further improvements in tasks requiring fine control. We propose a novel method for generating ancient Chinese lyrics (Song Ci), a type of ancient lyrics that involves precise control of song structure. The system is equipped with a phrase retriever and a phrase connector. Based on an input prompt, the phrase retriever picks phrases from a database to construct a phrase pool. The phrase connector then selects a series of phrases from the phrase pool that minimizes a multi-term loss function that considers rhyme, song structure, and fluency. Experimental results show that our method can generate high-quality ancient Chinese lyrics while performing well on topic and song structure control. We also expect our approach to be generalized to other lyrics-generating tasks.
Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving
Venue: arXiv cs.CV 2023
Year: 2023

3D object detection is an important task in autonomous driving to perceive the surroundings. Despite the excellent performance, the existing 3D detectors lack the robustness to real-world corruptions caused by adverse weathers, sensor noises, etc., provoking concerns about the safety and reliability of autonomous driving systems. To comprehensively and rigorously benchmark the corruption robustness of 3D detectors, in this paper we design 27 types of common corruptions for both LiDAR and camera inputs considering real-world driving scenarios. By synthesizing these corruptions on public datasets, we establish three corruption robustness benchmarks -- KITTI-C, nuScenes-C, and Waymo-C. Then, we conduct large-scale experiments on 24 diverse 3D object detection models to evaluate their corruption robustness. Based on the evaluation results, we draw several important findings, including: 1) motion-level corruptions are the most threatening ones that lead to significant performance drop of all models; 2) LiDAR-camera fusion models demonstrate better robustness; 3) camera-only models are extremely vulnerable to image corruptions, showing the indispensability of LiDAR point clouds. We release the benchmarks and codes at https://github.com/kkkcx/3D_Corruptions_AD. We hope that our benchmarks and findings can provide insights for future research on developing robust 3D object detection models.
From Sparse to Precise: A Practical Editing Approach for Intracardiac Echocardiography Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

Accurate and safe catheter ablation procedures for patients with atrial fibrillation require precise segmentation of cardiac structures in Intracardiac Echocardiography (ICE) imaging. Prior studies have suggested methods that employ 3D geometry information from the ICE transducer to create a sparse ICE volume by placing 2D frames in a 3D grid, enabling training of 3D segmentation models. However, the resulting 3D masks from these models can be inaccurate and may lead to serious clinical complications due to the sparse sampling in ICE data, frames misalignment, and cardiac motion. To address this issue, we propose an interactive editing framework that allows users to edit segmentation output by drawing scribbles on a 2D frame. The user interaction is mapped to the 3D grid and utilized to execute an editing step that modifies the segmentation in the vicinity of the interaction while preserving the previous segmentation away from the interaction. Furthermore, our framework accommodates multiple edits to the segmentation output in a sequential manner without compromising previous edits. This paper presents a novel loss function and a novel evaluation metric specifically designed for editing. Results from cross-validation and testing indicate that our proposed loss function outperforms standard losses and training strategies in terms of segmentation quality and following user input. Additionally, we show quantitatively and qualitatively that subsequent edits do not compromise previous edits when using our method, as opposed to standard segmentation losses. Overall, our approach enhances the accuracy of the segmentation while avoiding undesired changes away from user interactions and without compromising the quality of previously edited regions, leading to better patient outcomes.
Hospitalization Length of Stay Prediction using Patient Event Sequences
Venue: arXiv cs.LG 2023
Year: 2023

Predicting patients hospital length of stay (LOS) is essential for improving resource allocation and supporting decision-making in healthcare organizations. This paper proposes a novel approach for predicting LOS by modeling patient information as sequences of events. Specifically, we present a transformer-based model, termed Medic-BERT (M-BERT), for LOS prediction using the unique features describing patients medical event sequences. We performed empirical experiments on a cohort of more than 45k emergency care patients from a large Danish hospital. Experimental results show that M-BERT can achieve high accuracy on a variety of LOS problems and outperforms traditional nonsequence-based machine learning approaches.
SoK: Scalability Techniques for BFT Consensus
Venue: arXiv cs.DC 2023
Year: 2023

With the advancement of blockchain systems, many recent research works have proposed distributed ledger technology~(DLT) that employs Byzantine fault-tolerant~(BFT) consensus protocols to decide which block to append next to the ledger. Notably, BFT consensus can offer high performance, energy efficiency, and provable correctness properties, and it is thus considered a promising building block for creating highly resilient and performant blockchain infrastructures. Yet, a major ongoing challenge is to make BFT consensus applicable to large-scale environments. A large body of recent work addresses this challenge by developing novel ideas to improve the scalability of BFT consensus, thus opening the path for a new generation of BFT protocols tailored to the needs of blockchain. In this survey, we create a systematization of knowledge about the novel scalability-enhancing techniques that state-of-the-art BFT consensus protocols use. For our comparison, we closely analyze the efforts, assumptions, and trade-offs these protocols make.
From MNIST to ImageNet and Back: Benchmarking Continual Curriculum Learning
Venue: arXiv cs.LG 2023
Year: 2023

Continual learning (CL) is one of the most promising trends in recent machine learning research. Its goal is to go beyond classical assumptions in machine learning and develop models and learning strategies that present high robustness in dynamic environments. The landscape of CL research is fragmented into several learning evaluation protocols, comprising different learning tasks, datasets, and evaluation metrics. Additionally, the benchmarks adopted so far are still distant from the complexity of real-world scenarios, and are usually tailored to highlight capabilities specific to certain strategies. In such a landscape, it is hard to objectively assess strategies. In this work, we fill this gap for CL on image data by introducing two novel CL benchmarks that involve multiple heterogeneous tasks from six image datasets, with varying levels of complexity and quality. Our aim is to fairly evaluate current state-of-the-art CL strategies on a common ground that is closer to complex real-world scenarios. We additionally structure our benchmarks so that tasks are presented in increasing and decreasing order of complexity -- according to a curriculum -- in order to evaluate if current CL models are able to exploit structure across tasks. We devote particular emphasis to providing the CL community with a rigorous and reproducible evaluation protocol for measuring the ability of a model to generalize and not to forget while learning. Furthermore, we provide an extensive experimental evaluation showing that popular CL strategies, when challenged with our benchmarks, yield sub-par performance, high levels of forgetting, and present a limited ability to effectively leverage curriculum task ordering. We believe that these results highlight the need for rigorous comparisons in future CL works as well as pave the way to design new CL strategies that are able to deal with more complex scenarios.
Runtime-Adaptable Selective Performance Instrumentation
Venue: arXiv cs.PF 2023
Year: 2023

Automated code instrumentation, i.e. the insertion of measurement hooks into a target application by the compiler, is an established technique for collecting reliable, fine-grained performance data. The set of functions to instrument has to be selected with care, as instrumenting every available function typically yields too large a runtime overhead, thus skewing the measurement. No "one-suits-all" selection mechanism exists, since the instrumentation decision is dependent on the measurement objective, the limit for tolerable runtime overhead and peculiarities of the target application. The Compiler-assisted Performance Instrumentation (CaPI) tool assists in creating such instrumentation configurations, by enabling the user to combine different selection mechanisms as part of a configurable selection pipeline, operating on a statically constructed whole-program call-graph. Previously, CaPI relied on a static instrumentation workflow which made the process of refining the initial selection quite cumbersome for large-scale codes, as the application had to be recompiled after each adjustment. In this work, we present new runtime-adaptable instrumentation capabilities for CaPI which do not require recompilation when instrumentation changes are made. To this end, the XRay instrumentation feature of the LLVM compiler was extended to support the instrumentation of shared dynamic objects. An XRay-compatible runtime system was added to CaPI that instruments selected functions at program start, thereby significantly reducing the required time for selection refinements. Furthermore, an interface to the TALP tool for recording parallel efficiency metrics was implemented, alongside a specialized selection module for creating suitable coarse-grained region instrumentations.