OpenRaiser/Intern-Atlas huggingface dataset:
Multi-domain learning CNN model for microscopy image classification
Venue: arXiv cs.CV 2023
Year: 2023

For any type of microscopy image, getting a deep learning model to work well requires considerable effort to select a suitable architecture and time to train it. As there is a wide range of microscopes and experimental setups, designing a single model that can apply to multiple imaging domains, instead of having multiple per-domain models, becomes more essential. This task is challenging and somehow overlooked in the literature. In this paper, we present a multi-domain learning architecture for the classification of microscopy images that differ significantly in types and contents. Unlike previous methods that are computationally intensive, we have developed a compact model, called Mobincep, by combining the simple but effective techniques of depth-wise separable convolution and the inception module. We also introduce a new optimization technique to regulate the latent feature space during training to improve the network's performance. We evaluated our model on three different public datasets and compared its performance in single-domain and multiple-domain learning modes. The proposed classifier surpasses state-of-the-art results and is robust for limited labeled data. Moreover, it helps to eliminate the burden of designing a new network when switching to new experiments.
NFT Marketplace
Venue: arXiv cs.MM 2023
Year: 2023

In an increasingly digitized world, the secure management and trade of digital assets have become a pressing issue. This project aims to address this challenge by developing a decentralized application (dApp) that leverages blockchain technology and deep learning models to provide secure and efficient digital asset management, with a focus on NFTs. The dApp includes features such as secure wallet connections, NFT image generation, minting, marketplace, and profile management. The back-end of the dApp is implemented using the Goerli testnet with Solidity-based smart contracts, while IPFS and ReactJS/EtherJS are used for decentralized storage and front-end development, respectively. Additionally, the OpenAI API is integrated to generate unique NFT images based on user input. The project demonstrates the practical application of blockchain technology and deep learning models in developing dApps for secure and decentralized digital asset management. Overall, the project contributes to the ongoing research on blockchain-based solutions for secure digital asset management, while highlighting the potential of blockchain and deep learning technologies to transform the way we manage and trade digital assets.
Meta Semantics: Towards better natural language understanding and reasoning
Venue: arXiv cs.CL 2023
Year: 2023

Natural language understanding is one of the most challenging topics in artificial intelligence. Deep neural network methods, particularly large language module (LLM) methods such as ChatGPT and GPT-3, have powerful flexibility to adopt informal text but are weak on logical deduction and suffer from the out-of-vocabulary (OOV) problem. On the other hand, rule-based methods such as Mathematica, Semantic web, and Lean, are excellent in reasoning but cannot handle the complex and changeable informal text. Inspired by pragmatics and structuralism, we propose two strategies to solve the OOV problem and a semantic model for better natural language understanding and reasoning.
A Plug-and-Play Defensive Perturbation for Copyright Protection of DNN-based Applications
Venue: arXiv cs.CV 2023
Year: 2023

Wide deployment of deep neural networks (DNNs) based applications (e.g., style transfer, cartoonish), stimulating the requirement of copyright protection of such application's production. Although some traditional visible copyright techniques are available, they would introduce undesired traces and result in a poor user experience. In this paper, we propose a novel plug-and-play invisible copyright protection method based on defensive perturbation for DNN-based applications (i.e., style transfer). Rather than apply the perturbation to attack the DNNs model, we explore the potential utilization of perturbation in copyright protection. Specifically, we project the copyright information to the defensive perturbation with the designed copyright encoder, which is added to the image to be protected. Then, we extract the copyright information from the encoded copyrighted image with the devised copyright decoder. Furthermore, we use a robustness module to strengthen the decoding capability of the decoder toward images with various distortions (e.g., JPEG compression), which may be occurred when the user posts the image on social media. To ensure the image quality of encoded images and decoded copyright images, a loss function was elaborately devised. Objective and subjective experiment results demonstrate the effectiveness of the proposed method. We have also conducted physical world tests on social media (i.e., Wechat and Twitter) by posting encoded copyright images. The results show that the copyright information in the encoded image saved from social media can still be correctly extracted.
Interactive System-wise Anomaly Detection
Venue: arXiv cs.LG 2023
Year: 2023

Anomaly detection, where data instances are discovered containing feature patterns different from the majority, plays a fundamental role in various applications. However, it is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. Appropriate interactions are needed to interact with the systems and identify those with abnormal responses. Detecting system-wise anomalies is a challenging task due to several reasons including: how to formally define the system-wise anomaly detection problem; how to find the effective activation signal for interacting with systems to progressively collect the data and learn the detector; how to guarantee stable training in such a non-stationary scenario with real-time interactions? To address the challenges, we propose InterSAD (Interactive System-wise Anomaly Detection). Specifically, first, we adopt Markov decision process to model the interactive systems, and define anomalous systems as anomalous transition and anomalous reward systems. Then, we develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings, and a policy network to generate effective activation for separating embeddings of normal and anomaly systems. Finally, we design a training method to stabilize the learning process, which includes a replay buffer to store historical interaction data and allow them to be re-sampled. Experiments on two benchmark environments, including identifying the anomalous robotic systems and detecting user data poisoning in recommendation models, demonstrate the superiority of InterSAD compared with state-of-the-art baselines methods.
Picking Up Quantization Steps for Compressed Image Classification
Venue: arXiv cs.CV 2023
Year: 2023

The sensitivity of deep neural networks to compressed images hinders their usage in many real applications, which means classification networks may fail just after taking a screenshot and saving it as a compressed file. In this paper, we argue that neglected disposable coding parameters stored in compressed files could be picked up to reduce the sensitivity of deep neural networks to compressed images. Specifically, we resort to using one of the representative parameters, quantization steps, to facilitate image classification. Firstly, based on quantization steps, we propose a novel quantization aware confidence (QAC), which is utilized as sample weights to reduce the influence of quantization on network training. Secondly, we utilize quantization steps to alleviate the variance of feature distributions, where a quantization aware batch normalization (QABN) is proposed to replace batch normalization of classification networks. Extensive experiments show that the proposed method significantly improves the performance of classification networks on CIFAR-10, CIFAR-100, and ImageNet. The code is released on https://github.com/LiMaPKU/QSAM.git
Natural Evolution Strategy for Mixed-Integer Black-Box Optimization
Venue: arXiv cs.NE 2023
Year: 2023

This paper proposes a natural evolution strategy (NES) for mixed-integer black-box optimization (MI-BBO) that appears in real-world problems such as hyperparameter optimization of machine learning and materials design. This problem is difficult to optimize because plateaus where the values do not change appear when the integer variables are relaxed to the continuous ones. CMA-ES w. Margin that addresses the plateaus reportedly showed good performance on MI-BBO benchmark problems. However, it has been observed that the search performance of CMA-ES w. Margin deteriorates when continuous variables contribute more to the objective function value than integer ones. In order to address the problem of CMA-ES w. Margin, we propose Distance-weighted eXponential Natural Evolution Strategy taking account of Implicit Constraint and Integer (DX-NES-ICI). We compare the search performance of DX-NES-ICI with that of CMA-ES w. Margin through numerical experiments. As a result, DX-NES-ICI was up to 3.7 times better than CMA-ES w. Margin in terms of a rate of finding the optimal solutions on benchmark problems where continuous variables contribute more to the objective function value than integer ones. DX-NES-ICI also outperformed CMA-ES w. Margin on problems where CMA-ES w. Margin originally showed good performance.
Smart Learning to Find Dumb Contracts (Extended Version)
Venue: arXiv cs.CR 2023
Year: 2023

We introduce the Deep Learning Vulnerability Analyzer (DLVA) for Ethereum smart contracts based on neural networks. We train DLVA to judge bytecode even though the supervising oracle can only judge source. DLVA's training algorithm is general: we extend a source code analysis to bytecode without any manual feature engineering, predefined patterns, or expert rules. DLVA's training algorithm is also robust: it overcame a 1.25% error rate mislabeled contracts, and--the student surpassing the teacher--found vulnerable contracts that Slither mislabeled. DLVA is much faster than other smart contract vulnerability detectors: DLVA checks contracts for 29 vulnerabilities in 0.2 seconds, a 10-1,000x speedup. DLVA has three key components. First, Smart Contract to Vector (SC2V) uses neural networks to map smart contract bytecode to a high-dimensional floating-point vector. We benchmark SC2V against 4 state-of-the-art graph neural networks and show that it improves model differentiation by 2.2%. Second, Sibling Detector (SD) classifies contracts when a target contract's vector is Euclidian-close to a labeled contract's vector in a training set; although only able to judge 55.7% of the contracts in our test set, it has a Slither-predictive accuracy of 97.4% with a false positive rate of only 0.1%. Third, Core Classifier (CC) uses neural networks to infer vulnerable contracts regardless of vector distance. We benchmark DLVA's CC with 10 ML techniques and show that the CC improves accuracy by 11.3%. Overall, DLVA predicts Slither's labels with an overall accuracy of 92.7% and associated false positive rate of 7.2%. Lastly, we benchmark DLVA against nine well-known smart contract analysis tools. Despite using much less analysis time, DLVA completed every query, leading the pack with an average accuracy of 99.7%, pleasingly balancing high true positive rates with low false positive rates.
Energy management system for biological 3D printing by the refinement of manifold model morphing in flexible grasping space
Venue: arXiv cs.GR 2023
Year: 2023

The use of 3D printing, or additive manufacturing, has gained significant attention in recent years due to its potential for revolutionizing traditional manufacturing processes. One key challenge in 3D printing is managing energy consumption, as it directly impacts the cost, efficiency, and sustainability of the process. In this paper, we propose an energy management system that leverages the refinement of manifold model morphing in a flexible grasping space, to reduce costs for biological 3D printing. The manifold model is a mathematical representation of the 3D object to be printed, and the refinement process involves optimizing the morphing parameters of the manifold model to achieve desired printing outcomes. To enable flexibility in the grasping space, we incorporate data-driven approaches, such as machine learning and data augmentation techniques, to enhance the accuracy and robustness of the energy management system. Our proposed system addresses the challenges of limited sample data and complex morphologies of manifold models in layered additive manufacturing. Our method is more applicable for soft robotics and biomechanisms. We evaluate the performance of our system through extensive experiments and demonstrate its effectiveness in predicting and managing energy consumption in 3D printing processes. The results highlight the importance of refining manifold model morphing in the flexible grasping space for achieving energy-efficient 3D printing, contributing to the advancement of green and sustainable manufacturing practices.
Multi-scale Evolutionary Neural Architecture Search for Deep Spiking Neural Networks
Venue: arXiv cs.NE 2023
Year: 2023

Spiking Neural Networks (SNNs) have received considerable attention not only for their superiority in energy efficiency with discrete signal processing but also for their natural suitability to integrate multi-scale biological plasticity. However, most SNNs directly adopt the structure of the well-established Deep Neural Networks (DNNs), and rarely automatically design Neural Architecture Search (NAS) for SNNs. The neural motifs topology, modular regional structure and global cross-brain region connection of the human brain are the product of natural evolution and can serve as a perfect reference for designing brain-inspired SNN architecture. In this paper, we propose a Multi-Scale Evolutionary Neural Architecture Search (MSE-NAS) for SNN, simultaneously considering micro-, meso- and macro-scale brain topologies as the evolutionary search space. MSE-NAS evolves individual neuron operation, self-organized integration of multiple circuit motifs, and global connectivity across motifs through a brain-inspired indirect evaluation function, Representational Dissimilarity Matrices (RDMs). This training-free fitness function could greatly reduce computational consumption and NAS's time, and its task-independent property enables the searched SNNs to exhibit excellent transferability on multiple datasets. Furthermore, MSE-NAS show robustness against the training method and noise. Extensive experiments demonstrate that the proposed algorithm achieves state-of-the-art (SOTA) performance with shorter simulation steps on static datasets (CIFAR10, CIFAR100) and neuromorphic datasets (CIFAR10-DVS and DVS128-Gesture). The thorough analysis also illustrates the significant performance improvement and consistent bio-interpretability deriving from the topological evolution at different scales and the RDMs fitness function.
Hyperbolic Geometry in Computer Vision: A Survey
Venue: arXiv cs.CV 2023
Year: 2023

Hyperbolic geometry, a Riemannian manifold endowed with constant sectional negative curvature, has been considered an alternative embedding space in many learning scenarios, \eg, natural language processing, graph learning, \etc, as a result of its intriguing property of encoding the data's hierarchical structure (like irregular graph or tree-likeness data). Recent studies prove that such data hierarchy also exists in the visual dataset, and investigate the successful practice of hyperbolic geometry in the computer vision (CV) regime, ranging from the classical image classification to advanced model adaptation learning. This paper presents the first and most up-to-date literature review of hyperbolic spaces for CV applications. To this end, we first introduce the background of hyperbolic geometry, followed by a comprehensive investigation of algorithms, with geometric prior of hyperbolic space, in the context of visual applications. We also conclude this manuscript and identify possible future directions.
Reddit in the Time of COVID
Venue: arXiv cs.CY 2023
Year: 2023

When the COVID-19 pandemic hit, much of life moved online. Platforms of all types reported surges of activity, and people remarked on the various important functions that online platforms suddenly fulfilled. However, researchers lack a rigorous understanding of the pandemic's impacts on social platforms, and whether they were temporary or long-lasting. We present a conceptual framework for studying the large-scale evolution of social platforms and apply it to the study of Reddit's history, with a particular focus on the COVID-19 pandemic. We study platform evolution through two key dimensions: structure vs. content and macro- vs. micro-level analysis. Structural signals help us quantify how much behavior changed, while content analysis clarifies exactly how it changed. Applying these at the macro-level illuminates platform-wide changes, while at the micro-level we study impacts on individual users. We illustrate the value of this approach by showing the extraordinary and ordinary changes Reddit went through during the pandemic. First, we show that typically when rapid growth occurs, it is driven by a few concentrated communities and within a narrow slice of language use. However, Reddit's growth throughout COVID-19 was spread across disparate communities and languages. Second, all groups were equally affected in their change of interest, but veteran users tended to invoke COVID-related language more than newer users. Third, the new wave of users that arrived following COVID-19 was fundamentally different from previous cohorts of new users in terms of interests, activity, and likelihood of staying active on the platform. These findings provide a more rigorous understanding of how an online platform changed during the global pandemic.
Eyettention: An Attention-based Dual-Sequence Model for Predicting Human Scanpaths during Reading
Venue: arXiv cs.CL 2023
Year: 2023

Eye movements during reading offer insights into both the reader's cognitive processes and the characteristics of the text that is being read. Hence, the analysis of scanpaths in reading have attracted increasing attention across fields, ranging from cognitive science over linguistics to computer science. In particular, eye-tracking-while-reading data has been argued to bear the potential to make machine-learning-based language models exhibit a more human-like linguistic behavior. However, one of the main challenges in modeling human scanpaths in reading is their dual-sequence nature: the words are ordered following the grammatical rules of the language, whereas the fixations are chronologically ordered. As humans do not strictly read from left-to-right, but rather skip or refixate words and regress to previous words, the alignment of the linguistic and the temporal sequence is non-trivial. In this paper, we develop Eyettention, the first dual-sequence model that simultaneously processes the sequence of words and the chronological sequence of fixations. The alignment of the two sequences is achieved by a cross-sequence attention mechanism. We show that Eyettention outperforms state-of-the-art models in predicting scanpaths. We provide an extensive within- and across-data set evaluation on different languages. An ablation study and qualitative analysis support an in-depth understanding of the model's behavior.
Individual Fairness in Bayesian Neural Networks
Venue: arXiv cs.LG 2023
Year: 2023

We study Individual Fairness (IF) for Bayesian neural networks (BNNs). Specifically, we consider the $ε$-$δ$-individual fairness notion, which requires that, for any pair of input points that are $ε$-similar according to a given similarity metrics, the output of the BNN is within a given tolerance $δ>0.$ We leverage bounds on statistical sampling over the input space and the relationship between adversarial robustness and individual fairness to derive a framework for the systematic estimation of $ε$-$δ$-IF, designing Fair-FGSM and Fair-PGD as global,fairness-aware extensions to gradient-based attacks for BNNs. We empirically study IF of a variety of approximately inferred BNNs with different architectures on fairness benchmarks, and compare against deterministic models learnt using frequentist techniques. Interestingly, we find that BNNs trained by means of approximate Bayesian inference consistently tend to be markedly more individually fair than their deterministic counterparts.
A Comprehensive Review on Ontologies for Scenario-based Testing in the Context of Autonomous Driving
Venue: arXiv cs.RO 2023
Year: 2023

The verification and validation of autonomous driving vehicles remains a major challenge due to the high complexity of autonomous driving functions. Scenario-based testing is a promising method for validating such a complex system. Ontologies can be utilized to produce test scenarios that are both meaningful and relevant. One crucial aspect of this process is selecting the appropriate method for describing the entities involved. The level of detail and specific entity classes required will vary depending on the system being tested. It is important to choose an ontology that properly reflects these needs. This paper summarizes key representative ontologies for scenario-based testing and related use cases in the field of autonomous driving. The considered ontologies are classified according to their level of detail for both static facts and dynamic aspects. Furthermore, the ontologies are evaluated based on the presence of important entity classes and the relations between them.
On the Importance of Exploration for Real Life Learned Algorithms
Venue: arXiv cs.LG 2023
Year: 2023

The quality of data driven learning algorithms scales significantly with the quality of data available. One of the most straight-forward ways to generate good data is to sample or explore the data source intelligently. Smart sampling can reduce the cost of gaining samples, reduce computation cost in learning, and enable the learning algorithm to adapt to unforeseen events. In this paper, we teach three Deep Q-Networks (DQN) with different exploration strategies to solve a problem of puncturing ongoing transmissions for URLLC messages. We demonstrate the efficiency of two adaptive exploration candidates, variance-based and Maximum Entropy-based exploration, compared to the standard, simple epsilon-greedy exploration approach.
Learning Robust, Agile, Natural Legged Locomotion Skills in the Wild
Venue: arXiv cs.RO 2023
Year: 2023

Recently, reinforcement learning has become a promising and polular solution for robot legged locomotion. Compared to model-based control, reinforcement learning based controllers can achieve better robustness against uncertainties of environments through sim-to-real learning. However, the corresponding learned gaits are in general overly conservative and unatural. In this paper, we propose a new framework for learning robust, agile and natural legged locomotion skills over challenging terrain. We incorporate an adversarial training branch based on real animal locomotion data upon a teacher-student training pipeline for robust sim-to-real transfer. Empirical results on both simulation and real world of a quadruped robot demonstrate that our proposed algorithm enables robustly traversing challenging terrains such as stairs, rocky ground and slippery floor with only proprioceptive perception. Meanwhile, the gaits are more agile, natural, and energy efficient compared to the baselines. Both qualitative and quantitative results are presented in this paper.