OpenRaiser/Intern-Atlas huggingface dataset:
SO-NeRF: Active View Planning for NeRF using Surrogate Objectives
Venue: arXiv cs.CV 2023
Year: 2023

Despite the great success of Neural Radiance Fields (NeRF), its data-gathering process remains vague with only a general rule of thumb of sampling as densely as possible. The lack of understanding of what actually constitutes good views for NeRF makes it difficult to actively plan a sequence of views that yield the maximal reconstruction quality. We propose Surrogate Objectives for Active Radiance Fields (SOAR), which is a set of interpretable functions that evaluates the goodness of views using geometric and photometric visual cues - surface coverage, geometric complexity, textural complexity, and ray diversity. Moreover, by learning to infer the SOAR scores from a deep network, SOARNet, we are able to effectively select views in mere seconds instead of hours, without the need for prior visits to all the candidate views or training any radiance field during such planning. Our experiments show SOARNet outperforms the baselines with $\sim$80x speed-up while achieving better or comparable reconstruction qualities. We finally show that SOAR is model-agnostic, thus it generalizes across fully neural-implicit to fully explicit approaches.
Monte Carlo Optimization for Solving Multilevel Stackelberg Games
Venue: arXiv cs.GT 2023
Year: 2023

Stackelberg games originate where there are market leaders and followers, and the actions of leaders influence the behavior of the followers. Mathematical modelling of such games results in what's called a Bilevel Optimization problem. There is an entire area of research dedicated to analyzing and solving Bilevel Optimization problems which are often complex, and finding solutions for such problems is known to be NP-Hard. A generalization of Stackelberg games is a Multilevel Stackelberg game where we may have nested leaders and followers, such that a follower is, in turn, a leader for all lower-level players. These problems are much more difficult to solve, and existing solution approaches typically require extensive cooperation between the players (which generally can't be assumed) or make restrictive assumptions about the structure of the problem. In this paper, we present a stochastic algorithm to approximate the local equilibrium solutions for these Multilevel games. We then construct a few examples of such Multilevel problems, including: a) a nested toll-setting problem; and b) an adversarial initial condition determination problem for Robust Trajectory Optimization. We test our algorithm on our constructed problems as well as some trilevel problems from the literature, and show that it is able to approximate the optimum solutions for these problems within a reasonable error margin. We also provide an asymptotic proof for the convergence of the algorithm and empirically analyze its accuracy and convergence speed for different parameters. Lastly, we compare it with existing solution strategies from the literature and demonstrate that it outperforms them.
STEP CATFormer: Spatial-Temporal Effective Body-Part Cross Attention Transformer for Skeleton-based Action Recognition
Venue: arXiv cs.CV 2023
Year: 2023

Graph convolutional networks (GCNs) have been widely used and achieved remarkable results in skeleton-based action recognition. We think the key to skeleton-based action recognition is a skeleton hanging in frames, so we focus on how the Graph Convolutional Convolution networks learn different topologies and effectively aggregate joint features in the global temporal and local temporal. In this work, we propose three Channel-wise Tolopogy Graph Convolution based on Channel-wise Topology Refinement Graph Convolution (CTR-GCN). Combining CTR-GCN with two joint cross-attention modules can capture the upper-lower body part and hand-foot relationship skeleton features. After that, to capture features of human skeletons changing in frames we design the Temporal Attention Transformers to extract skeletons effectively. The Temporal Attention Transformers can learn the temporal features of human skeleton sequences. Finally, we fuse the temporal features output scale with MLP and classification. We develop a powerful graph convolutional network named Spatial Temporal Effective Body-part Cross Attention Transformer which notably high-performance on the NTU RGB+D, NTU RGB+D 120 datasets. Our code and models are available at https://github.com/maclong01/STEP-CATFormer
Masking Behaviors in Epidemiological Networks with Cognitively-plausible Reinforcement Learning
Venue: arXiv cs.SI 2023
Year: 2023

The COVID-19 pandemic highlighted the critical role of human behavior in influencing infectious disease transmission and the need for models capturing this complex dynamic. We present an agent-based model integrating an epidemiological simulation of disease spread with a cognitive architecture driving individual mask-wearing decisions. Agents decide whether to mask based on a utility function weighting factors like peer conformity, personal risk tolerance, and mask-wearing discomfort. By conducting experiments systematically varying behavioral model parameters and social network structures, we demonstrate how adaptive decision-making interacts with network connectivity patterns to impact population-level infection outcomes. The model provides a flexible computational framework for gaining insights into how behavioral interventions like mask mandates may differentially influence disease spread across communities with diverse social structures. Findings highlight the importance of integrating realistic human decision processes in epidemiological models to inform policy decisions during public health crises.
A path forward: Improving Internet routing security by enabling zones of trust
Venue: arXiv cs.NI 2023
Year: 2023

Although Internet routing security best practices have recently seen auspicious increases in uptake, ISPs have limited incentives to deploy them. They are operationally complex and expensive to implement, provide little competitive advantage, and protect only against origin hijacks, leaving unresolved the more general threat of path hijacks. We propose a new approach that achieves four design goals: improved incentive alignment to implement best practices; protection against path hijacks; expanded scope of such protection to customers of those engaged in the practices; and reliance on existing capabilities rather than needing complex new software in every participating router. Our proposal leverages an existing coherent core of interconnected ISPs to create a zone of trust, a topological region that protects not only all networks in the region, but all directly attached customers of those networks. Customers benefit from choosing ISPs committed to the practices, and ISPs thus benefit from committing to the practices. We compare our approach to other schemes, and discuss how a related proposal, ASPA, could be used to increase the scope of protection our scheme achieves. We hope this proposal inspires discussion of how the industry can make practical, measurable progress against the threat of route hijacks in the short term by leveraging institutionalized cooperation rooted in transparency and accountability.
PointMoment:Mixed-Moment-based Self-Supervised Representation Learning for 3D Point Clouds
Venue: arXiv cs.CV 2023
Year: 2023

Large and rich data is a prerequisite for effective training of deep neural networks. However, the irregularity of point cloud data makes manual annotation time-consuming and laborious. Self-supervised representation learning, which leverages the intrinsic structure of large-scale unlabelled data to learn meaningful feature representations, has attracted increasing attention in the field of point cloud research. However, self-supervised representation learning often suffers from model collapse, resulting in reduced information and diversity of the learned representation, and consequently degrading the performance of downstream tasks. To address this problem, we propose PointMoment, a novel framework for point cloud self-supervised representation learning that utilizes a high-order mixed moment loss function rather than the conventional contrastive loss function. Moreover, our framework does not require any special techniques such as asymmetric network architectures, gradient stopping, etc. Specifically, we calculate the high-order mixed moment of the feature variables and force them to decompose into products of their individual moment, thereby making multiple variables more independent and minimizing the feature redundancy. We also incorporate a contrastive learning approach to maximize the feature invariance under different data augmentations of the same point cloud. Experimental results show that our approach outperforms previous unsupervised learning methods on the downstream task of 3D point cloud classification and segmentation.
Detecting Voice Cloning Attacks via Timbre Watermarking
Venue: arXiv cs.SD 2023
Year: 2023

Nowadays, it is common to release audio content to the public. However, with the rise of voice cloning technology, attackers have the potential to easily impersonate a specific person by utilizing his publicly released audio without any permission. Therefore, it becomes significant to detect any potential misuse of the released audio content and protect its timbre from being impersonated. To this end, we introduce a novel concept, "Timbre Watermarking", which embeds watermark information into the target individual's speech, eventually defeating the voice cloning attacks. To ensure the watermark is robust to the voice cloning model's learning process, we design an end-to-end voice cloning-resistant detection framework. The core idea of our solution is to embed and extract the watermark in the frequency domain in a temporally invariant manner. To acquire generalization across different voice cloning attacks, we modulate their shared process and integrate it into our framework as a distortion layer. Experiments demonstrate that the proposed timbre watermarking can defend against different voice cloning attacks, exhibit strong resistance against various adaptive attacks (e.g., reconstruction-based removal attacks, watermark overwriting attacks), and achieve practicality in real-world services such as PaddleSpeech, Voice-Cloning-App, and so-vits-svc. In addition, ablation studies are also conducted to verify the effectiveness of our design. Some audio samples are available at https://timbrewatermarking.github.io/samples.
Artist-Friendly Relightable and Animatable Neural Heads
Venue: arXiv cs.CV 2023
Year: 2023

An increasingly common approach for creating photo-realistic digital avatars is through the use of volumetric neural fields. The original neural radiance field (NeRF) allowed for impressive novel view synthesis of static heads when trained on a set of multi-view images, and follow up methods showed that these neural representations can be extended to dynamic avatars. Recently, new variants also surpassed the usual drawback of baked-in illumination in neural representations, showing that static neural avatars can be relit in any environment. In this work we simultaneously tackle both the motion and illumination problem, proposing a new method for relightable and animatable neural heads. Our method builds on a proven dynamic avatar approach based on a mixture of volumetric primitives, combined with a recently-proposed lightweight hardware setup for relightable neural fields, and includes a novel architecture that allows relighting dynamic neural avatars performing unseen expressions in any environment, even with nearfield illumination and viewpoints.
JAMMIN-GPT: Text-based Improvisation using LLMs in Ableton Live
Venue: arXiv cs.SD 2023
Year: 2023

We introduce a system that allows users of Ableton Live to create MIDI-clips by naming them with musical descriptions. Users can compose by typing the desired musical content directly in Ableton's clip view, which is then inserted by our integrated system. This allows users to stay in the flow of their creative process while quickly generating musical ideas. The system works by prompting ChatGPT to reply using one of several text-based musical formats, such as ABC notation, chord symbols, or drum tablature. This is an important step in integrating generative AI tools into pre-existing musical workflows, and could be valuable for content makers who prefer to express their creative vision through descriptive language. Code is available at https://github.com/supersational/JAMMIN-GPT.
Modeling Aggregate Downwash Forces for Dense Multirotor Flight
Venue: arXiv cs.RO 2023
Year: 2023

Dense formation flight with multirotor swarms is a powerful, nature-inspired flight regime with numerous applications in the realworld. However, when multirotors fly in close vertical proximity to each other, the propeller downwash from the vehicles can have a destabilising effect on each other. Unfortunately, even in a homogeneous team, an accurate model of downwash forces from one vehicle is unlikely to be sufficient for predicting aggregate forces from multiple vehicles in formation. In this work, we model the interaction patterns produced by one or more vehicles flying in close proximity to an ego-vehicle. We first present an experimental test rig designed to capture 6-DOF exogenic forces acting on a multirotor frame. We then study and characterize these measured forces as a function of the relative states of two multirotors flying various patterns in its vicinity. Our analysis captures strong non-linearities present in the aggregation of these interactions. Then, by modeling the formation as a graph, we present a novel approach for learning the force aggregation function, and contrast it against simpler linear models. Finally, we explore how our proposed models generalize when a fourth vehicle is added to the formation.
Towards Sobolev Pruning
Venue: arXiv cs.LG 2023
Year: 2023

The increasing use of stochastic models for describing complex phenomena warrants surrogate models that capture the reference model characteristics at a fraction of the computational cost, foregoing potentially expensive Monte Carlo simulation. The predominant approach of fitting a large neural network and then pruning it to a reduced size has commonly neglected shortcomings. The produced surrogate models often will not capture the sensitivities and uncertainties inherent in the original model. In particular, (higher-order) derivative information of such surrogates could differ drastically. Given a large enough network, we expect this derivative information to match. However, the pruned model will almost certainly not share this behavior. In this paper, we propose to find surrogate models by using sensitivity information throughout the learning and pruning process. We build on work using Interval Adjoint Significance Analysis for pruning and combine it with the recent advancements in Sobolev Training to accurately model the original sensitivity information in the pruned neural network based surrogate model. We experimentally underpin the method on an example of pricing a multidimensional Basket option modelled through a stochastic differential equation with Brownian motion. The proposed method is, however, not limited to the domain of quantitative finance, which was chosen as a case study for intuitive interpretations of the sensitivities. It serves as a foundation for building further surrogate modelling techniques considering sensitivity information.
Active Wildfires Detection and Dynamic Escape Routes Planning for Humans through Information Fusion between Drones and Satellites
Venue: arXiv cs.AI 2023
Year: 2023

UAVs are playing an increasingly important role in the field of wilderness rescue by virtue of their flexibility. This paper proposes a fusion of UAV vision technology and satellite image analysis technology for active wildfires detection and road networks extraction of wildfire areas and real-time dynamic escape route planning for people in distress. Firstly, the fire source location and the segmentation of smoke and flames are targeted based on Sentinel 2 satellite imagery. Secondly, the road segmentation and the road condition assessment are performed by D-linkNet and NDVI values in the central area of the fire source by UAV. Finally, the dynamic optimal route planning for humans in real time is performed by the weighted A* algorithm in the road network with the dynamic fire spread model. Taking the Chongqing wildfire on August 24, 2022, as a case study, the results demonstrate that the dynamic escape route planning algorithm can provide an optimal real-time navigation path for humans in the presence of fire through the information fusion of UAVs and satellites.
Incorporating the algorithm for the boundary condition from FVM into the framework of Eulerian SPH
Venue: arXiv cs.CE 2023
Year: 2023

Finite volume method (FVM) is a widely used mesh-based technique, renowned for its computational efficiency and accuracy but it bears significant drawbacks, particularly in mesh generation and handling complex boundary interfaces or conditions. On the other hand, smoothed particle hydrodynamics (SPH) method, a popular meshless alternative, inherently circumvents the mesh generation and yields smoother numerical outcomes but at the expense of computational efficiency. Therefore, numerous researchers have strategically amalgamated the strengths of both methods to investigate complex flow phenomena and this synergy has yielded precise and computationally efficient outcomes. However, algorithms involving the weak coupling of these two methods tend to be intricate, which has issues pertaining to versatility, implementation, and mutual adaptation to hardware and coding structures. Thus, achieving a robust and strong coupling of FVM and SPH in a unified framework is imperative. Due to differing boundary algorithms between these methods in Wang's work, the crucial step for establishing a strong coupling of both methods within a unified SPH framework lies in incorporating the FVM boundary algorithm into the Eulerian SPH method. In this paper, we propose a straightforward algorithm in the Eulerian SPH method, algorithmically equivalent to that in FVM, grounded in the principle of zero-order consistency. Moreover, several numerical examples, including fully and weakly compressible flows with various boundary conditions in the Eulerian SPH method, validate the stability and accuracy of the proposed algorithm.
Enhancing Kinship Verification through Multiscale Retinex and Combined Deep-Shallow features
Venue: arXiv cs.CV 2023
Year: 2023

The challenge of kinship verification from facial images represents a cutting-edge and formidable frontier in the realms of pattern recognition and computer vision. This area of study holds a myriad of potential applications, spanning from image annotation and forensic analysis to social media research. Our research stands out by integrating a preprocessing method named Multiscale Retinex (MSR), which elevates image quality and amplifies contrast, ultimately bolstering the end results. Strategically, our methodology capitalizes on the harmonious blend of deep and shallow texture descriptors, merging them proficiently at the score level through the Logistic Regression (LR) method. To elucidate, we employ the Local Phase Quantization (LPQ) descriptor to extract shallow texture characteristics. For deep feature extraction, we turn to the prowess of the VGG16 model, which is pre-trained on a convolutional neural network (CNN). The robustness and efficacy of our method have been put to the test through meticulous experiments on three rigorous kinship datasets, namely: Cornell Kin Face, UB Kin Face, and TS Kin Face.
Foundation Model Assisted Weakly Supervised Semantic Segmentation
Venue: arXiv cs.CV 2023
Year: 2023

This work aims to leverage pre-trained foundation models, such as contrastive language-image pre-training (CLIP) and segment anything model (SAM), to address weakly supervised semantic segmentation (WSSS) using image-level labels. To this end, we propose a coarse-to-fine framework based on CLIP and SAM for generating high-quality segmentation seeds. Specifically, we construct an image classification task and a seed segmentation task, which are jointly performed by CLIP with frozen weights and two sets of learnable task-specific prompts. A SAM-based seeding (SAMS) module is designed and applied to each task to produce either coarse or fine seed maps. Moreover, we design a multi-label contrastive loss supervised by image-level labels and a CAM activation loss supervised by the generated coarse seed map. These losses are used to learn the prompts, which are the only parts need to be learned in our framework. Once the prompts are learned, we input each image along with the learned segmentation-specific prompts into CLIP and the SAMS module to produce high-quality segmentation seeds. These seeds serve as pseudo labels to train an off-the-shelf segmentation network like other two-stage WSSS methods. Experiments show that our method achieves the state-of-the-art performance on PASCAL VOC 2012 and competitive results on MS COCO 2014. Code is available at https://github.com/HAL-42/FMA-WSSS.git.
Revisiting Micro and Macro Expressions in Computer Graphics Characters
Venue: arXiv cs.GR 2023
Year: 2023

This paper presents the reproduction of two studies focused on the perception of micro and macro expressions of Virtual Humans (VHs) generated by Computer Graphics (CG), first described in 2014 and replicated in 2021. The 2014 study referred to a VH realistic, whereas, in 2021, it referred to a VH cartoon. In our work, we replicate the study by using a realistic CG character. Our main goals are to compare the perceptions of micro and macro expressions between levels of realism (2021 cartoon versus 2023 realistic) and between realistic characters in different periods (i.e., 2014 versus 2023). In one of our results, people more easily recognized micro expressions in realistic VHs than in a cartoon VH. In another result, we show that the participants' perception was similar for both micro and macro expressions in 2014 and 2023.
Streaming Algorithms for the $k$-Submodular Cover Problem
Venue: arXiv cs.DS 2023
Year: 2023

Given a natural number $k\ge 2$, we consider the $k$-submodular cover problem ($k$-SC). The objective is to find a minimum cost subset of a ground set $\mathcal{X}$ subject to the value of a $k$-submodular utility function being at least a certain predetermined value $τ$. For this problem, we design a bicriteria algorithm with a cost at most $O(1/ε)$ times the optimal value, while the utility is at least $(1-ε)τ/r$, where $r$ depends on the monotonicity of $g$.
Augmenting optimization-based molecular design with graph neural networks
Venue: arXiv cs.CE 2023
Year: 2023

Computer-aided molecular design (CAMD) studies quantitative structure-property relationships and discovers desired molecules using optimization algorithms. With the emergence of machine learning models, CAMD score functions may be replaced by various surrogates to automatically learn the structure-property relationships. Due to their outstanding performance on graph domains, graph neural networks (GNNs) have recently appeared frequently in CAMD. But using GNNs introduces new optimization challenges. This paper formulates GNNs using mixed-integer programming and then integrates this GNN formulation into the optimization and machine learning toolkit OMLT. To characterize and formulate molecules, we inherit the well-established mixed-integer optimization formulation for CAMD and propose symmetry-breaking constraints to remove symmetric solutions caused by graph isomorphism. In two case studies, we investigate fragment-based odorant molecular design with more practical requirements to test the compatibility and performance of our approaches.