OpenRaiser/Intern-Atlas huggingface dataset:
RoMo: Robust Motion Segmentation Improves Structure from Motion
Venue: arXiv cs.CV 2024
Year: 2024

There has been extensive progress in the reconstruction and generation of 4D scenes from monocular casually-captured video. While these tasks rely heavily on known camera poses, the problem of finding such poses using structure-from-motion (SfM) often depends on robustly separating static from dynamic parts of a video. The lack of a robust solution to this problem limits the performance of SfM camera-calibration pipelines. We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. It outperforms unsupervised baselines for motion segmentation as well as supervised baselines trained from synthetic data. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.
HDI-Former: Hybrid Dynamic Interaction ANN-SNN Transformer for Object Detection Using Frames and Events
Venue: arXiv cs.CV 2024
Year: 2024

Combining the complementary benefits of frames and events has been widely used for object detection in challenging scenarios. However, most object detection methods use two independent Artificial Neural Network (ANN) branches, limiting cross-modality information interaction across the two visual streams and encountering challenges in extracting temporal cues from event streams with low power consumption. To address these challenges, we propose HDI-Former, a Hybrid Dynamic Interaction ANN-SNN Transformer, marking the first trial to design a directly trained hybrid ANN-SNN architecture for high-accuracy and energy-efficient object detection using frames and events. Technically, we first present a novel semantic-enhanced self-attention mechanism that strengthens the correlation between image encoding tokens within the ANN Transformer branch for better performance. Then, we design a Spiking Swin Transformer branch to model temporal cues from event streams with low power consumption. Finally, we propose a bio-inspired dynamic interaction mechanism between ANN and SNN sub-networks for cross-modality information interaction. The results demonstrate that our HDI-Former outperforms eleven state-of-the-art methods and our four baselines by a large margin. Our SNN branch also shows comparable performance to the ANN with the same architecture while consuming 10.57$\times$ less energy on the DSEC-Detection dataset. Our open-source code is available in the supplementary material.
Spatiotemporal Skip Guidance for Enhanced Video Diffusion Sampling
Venue: arXiv cs.CV 2024
Year: 2024

Diffusion models have emerged as a powerful tool for generating high-quality images, videos, and 3D content. While sampling guidance techniques like CFG improve quality, they reduce diversity and motion. Autoguidance mitigates these issues but demands extra weak model training, limiting its practicality for large-scale models. In this work, we introduce Spatiotemporal Skip Guidance (STG), a simple training-free sampling guidance method for enhancing transformer-based video diffusion models. STG employs an implicit weak model via self-perturbation, avoiding the need for external models or additional training. By selectively skipping spatiotemporal layers, STG produces an aligned, degraded version of the original model to boost sample quality without compromising diversity or dynamic degree. Our contributions include: (1) introducing STG as an efficient, high-performing guidance technique for video diffusion models, (2) eliminating the need for auxiliary models by simulating a weak model through layer skipping, and (3) ensuring quality-enhanced guidance without compromising sample diversity or dynamics unlike CFG. For additional results, visit https://junhahyung.github.io/STGuidance.
MatchDiffusion: Training-free Generation of Match-cuts
Venue: arXiv cs.CV 2024
Year: 2024

Match-cuts are powerful cinematic tools that create seamless transitions between scenes, delivering strong visual and metaphorical connections. However, crafting match-cuts is a challenging, resource-intensive process requiring deliberate artistic planning. In MatchDiffusion, we present the first training-free method for match-cut generation using text-to-video diffusion models. MatchDiffusion leverages a key property of diffusion models: early denoising steps define the scene's broad structure, while later steps add details. Guided by this insight, MatchDiffusion employs "Joint Diffusion" to initialize generation for two prompts from shared noise, aligning structure and motion. It then applies "Disjoint Diffusion", allowing the videos to diverge and introduce unique details. This approach produces visually coherent videos suited for match-cuts. User studies and metrics demonstrate MatchDiffusion's effectiveness and potential to democratize match-cut creation.
An indicator for effectiveness of text-to-image guardrails utilizing the Single-Turn Crescendo Attack (STCA)
Venue: arXiv cs.CR 2024
Year: 2024

The Single-Turn Crescendo Attack (STCA), first introduced in Aqrawi and Abbasi [2024], is an innovative method designed to bypass the ethical safeguards of text-to-text AI models, compelling them to generate harmful content. This technique leverages a strategic escalation of context within a single prompt, combined with trust-building mechanisms, to subtly deceive the model into producing unintended outputs. Extending the application of STCA to text-to-image models, we demonstrate its efficacy by compromising the guardrails of a widely-used model, DALL-E 3, achieving outputs comparable to outputs from the uncensored model Flux Schnell, which served as a baseline control. This study provides a framework for researchers to rigorously evaluate the robustness of guardrails in text-to-image models and benchmark their resilience against adversarial attacks.
Near-Optimal Trace Reconstruction for Mildly Separated Strings
Venue: arXiv cs.DS 2024
Year: 2024

In the trace reconstruction problem our goal is to learn an unknown string $x\in \{0,1\}^n$ given independent traces of $x$. A trace is obtained by independently deleting each bit of $x$ with some probability $δ$ and concatenating the remaining bits. It is a major open question whether the trace reconstruction problem can be solved with a polynomial number of traces when the deletion probability $δ$ is constant. The best known upper bound and lower bounds are respectively $\exp(\tilde O(n^{1/5}))$ and $\tilde Ω(n^{3/2})$ both by Chase [Cha21b,Cha21a]. Our main result is that if the string $x$ is mildly separated, meaning that the number of zeros between any two ones in $x$ is at least polylog$n$, and if $δ$ is a sufficiently small constant, then the trace reconstruction problem can be solved with $O(n \log n)$ traces and in polynomial time.
Sharing the Path: A Threshold Scheme from Isogenies and Error Correcting Codes
Venue: arXiv cs.CR 2024
Year: 2024

In 2022, a prominent supersingular isogeny-based cryptographic scheme, namely SIDH, was compromised by a key recovery attack. However, this attack does not undermine the isogeny path problem, which remains central to the security of isogeny-based cryptography. Following the attacks by Castryck and Decru, as well as Maino and Martindale, Robert gave a mature and polynomial-time algorithm that transforms the SIDH key recovery attack into a valuable cryptographic tool. In this paper, we combine this tool with advanced encoding techniques to construct a novel threshold scheme.
Near-Field Wideband Beamforming for RIS Based on Fresnel Zone
Venue: arXiv cs.IT 2024
Year: 2024

Reconfigurable intelligent surface (RIS) has emerged as a promising solution to overcome the challenges of high path loss and easy signal blockage in millimeter-wave (mmWave) and terahertz (THz) communication systems. With the increase of RIS aperture and system bandwidth, the near-field beam split effect emerges, which causes beams at different frequencies to focus on distinct physical locations, leading to a significant gain loss of beamforming. To address this problem, we leverage the property of Fresnel zone that the beam split disappears for RIS elements along a single Fresnel zone and propose beamforming design on the two dimensions of along and across the Fresnel zones. The phase shift of RIS elements along the same Fresnel zone are designed aligned, so that the signal reflected by these element can add up in-phase at the receiver regardless of the frequency. Then the expression of equivalent channel is simplified to the Fourier transform of reflective intensity across Fresnel zones modulated by the designed phase. Based on this relationship, we prove that the uniformly distributed in-band gain with aligned phase along the Fresnel zone leads to the upper bound of achievable rate. Finally, we design phase shifts of RIS to approach this upper bound by adopting the stationary phase method as well as the Gerchberg-Saxton (GS) algorithm. Simulation results validate the effectiveness of our proposed Fresnel zone-based method in mitigating the near-field beam split effect.
Data Augmentation with Diffusion Models for Colon Polyp Localization on the Low Data Regime: How much real data is enough?
Venue: arXiv cs.CV 2024
Year: 2024

The scarcity of data in medical domains hinders the performance of Deep Learning models. Data augmentation techniques can alleviate that problem, but they usually rely on functional transformations of the data that do not guarantee to preserve the original tasks. To approximate the distribution of the data using generative models is a way of reducing that problem and also to obtain new samples that resemble the original data. Denoising Diffusion models is a promising Deep Learning technique that can learn good approximations of different kinds of data like images, time series or tabular data. Automatic colonoscopy analysis and specifically Polyp localization in colonoscopy videos is a task that can assist clinical diagnosis and treatment. The annotation of video frames for training a deep learning model is a time consuming task and usually only small datasets can be obtained. The fine tuning of application models using a large dataset of generated data could be an alternative to improve their performance. We conduct a set of experiments training different diffusion models that can generate jointly colonoscopy images with localization annotations using a combination of existing open datasets. The generated data is used on various transfer learning experiments in the task of polyp localization with a model based on YOLO v9 on the low data regime.
Guardians of the Ledger: Protecting Decentralized Exchanges from State Derailment Defects
Venue: arXiv cs.SE 2024
Year: 2024

The decentralized exchange (DEX) leverages smart contracts to trade digital assets for users on the blockchain. Developers usually develop several smart contracts into one project, implementing complex logic functions and multiple transaction operations. However, the interaction among these contracts poses challenges for developers analyzing the state logic. Due to the complex state logic in DEX projects, many critical state derailment defects have emerged in recent years. In this paper, we conduct the first systematic study of state derailment defects in DEX. We define five categories of state derailment defects and provide detailed analyses of them. Furthermore, we propose a novel deep learning-based framework StateGuard for detecting state derailment defects in DEX smart contracts. It leverages a smart contract deconstructor to deconstruct the contract into an Abstract Syntax Tree (AST), from which five categories of dependency features are extracted. Next, it implements a graph optimizer to process the structured data. At last, the optimized data is analyzed by Graph Convolutional Networks (GCNs) to identify potential state derailment defects. We evaluated StateGuard through a dataset of 46 DEX projects containing 5,671 smart contracts, and it achieved 94.25% F1-score. In addition, in a comparison experiment with state-of-the-art, StateGuard leads the F1-score by 6.29%. To further verify its practicality, we used StateGuar to audit real-world contracts and successfully authenticated multiple novel CVEs.
Random Sampling for Diffusion-based Adversarial Purification
Venue: arXiv cs.CV 2024
Year: 2024

Denoising Diffusion Probabilistic Models (DDPMs) have gained great attention in adversarial purification. Current diffusion-based works focus on designing effective condition-guided mechanisms while ignoring a fundamental problem, i.e., the original DDPM sampling is intended for stable generation, which may not be the optimal solution for adversarial purification. Inspired by the stability of the Denoising Diffusion Implicit Model (DDIM), we propose an opposite sampling scheme called random sampling. In brief, random sampling will sample from a random noisy space during each diffusion process, while DDPM and DDIM sampling will continuously sample from the adjacent or original noisy space. Thus, random sampling obtains more randomness and achieves stronger robustness against adversarial attacks. Correspondingly, we also introduce a novel mediator conditional guidance to guarantee the consistency of the prediction under the purified image and clean image input. To expand awareness of guided diffusion purification, we conduct a detailed evaluation with different sampling methods and our random sampling achieves an impressive improvement in multiple settings. Leveraging mediator-guided random sampling, we also establish a baseline method named DiffAP, which significantly outperforms state-of-the-art (SOTA) approaches in performance and defensive stability. Remarkably, under strong attack, our DiffAP even achieves a more than 20% robustness advantage with 10$\times$ sampling acceleration.
Det-SAM2:Technical Report on the Self-Prompting Segmentation Framework Based on Segment Anything Model 2
Venue: arXiv cs.CV 2024
Year: 2024

Segment Anything Model 2 (SAM2) demonstrates exceptional performance in video segmentation and refinement of segmentation results. We anticipate that it can further evolve to achieve higher levels of automation for practical applications. Building upon SAM2, we conducted a series of practices that ultimately led to the development of a fully automated pipeline, termed Det-SAM2, in which object prompts are automatically generated by a detection model to facilitate inference and refinement by SAM2. This pipeline enables inference on infinitely long video streams with constant VRAM and RAM usage, all while preserving the same efficiency and accuracy as the original SAM2. This technical report focuses on the construction of the overall Det-SAM2 framework and the subsequent engineering optimization applied to SAM2. We present a case demonstrating an application built on the Det-SAM2 framework: AI refereeing in a billiards scenario, derived from our business context. The project at \url{https://github.com/motern88/Det-SAM2}.
The Complexity of Order-Finding for ROABPs
Venue: arXiv cs.CC 2024
Year: 2024

We study the \emph{order-finding problem} for Read-once Oblivious Algebraic Branching Programs (ROABPs). Given a polynomial $f$ and a parameter $w$, the goal is to find an order $σ$ in which $f$ has an ROABP of \emph{width} $w$. We show that this problem is NP-hard in the worst case, even when the input is a constant degree polynomial that is given in its dense representation. We provide a reduction from CutWidth to prove these results. Owing to the exactness of our reduction, all the known results for the hardness of approximation of Cutwidth also transfer directly to the order-finding problem. Additionally, we also show that any constant-approximation algorithm for the order-finding problem would imply a polynomial time approximation scheme (PTAS) for it. On the algorithmic front, we design algorithms that solve the order-finding problem for generic ROABPs in polynomial time, when the width $w$ is polynomial in the individual degree $d$ of the polynomial $f$. That is, our algorithm is efficient for most/random ROABPs, and requires more time only on a lower-dimensional subspace (or subvariety) of ROABPs. Even when the individual degree is constant, our algorithm runs in time $n^{O(\log w)}$ for most/random ROABPs. This stands in strong contrast to the case of (Boolean) ROBPs, where only heuristic order-finding algorithms are known.
SPAgent: Adaptive Task Decomposition and Model Selection for General Video Generation and Editing
Venue: arXiv cs.CV 2024
Year: 2024

While open-source video generation and editing models have made significant progress, individual models are typically limited to specific tasks, failing to meet the diverse needs of users. Effectively coordinating these models can unlock a wide range of video generation and editing capabilities. However, manual coordination is complex and time-consuming, requiring users to deeply understand task requirements and possess comprehensive knowledge of each model's performance, applicability, and limitations, thereby increasing the barrier to entry. To address these challenges, we propose a novel video generation and editing system powered by our Semantic Planning Agent (SPAgent). SPAgent bridges the gap between diverse user intents and the effective utilization of existing generative models, enhancing the adaptability, efficiency, and overall quality of video generation and editing. Specifically, the SPAgent assembles a tool library integrating state-of-the-art open-source image and video generation and editing models as tools. After fine-tuning on our manually annotated dataset, SPAgent can automatically coordinate the tools for video generation and editing, through our novelly designed three-step framework: (1) decoupled intent recognition, (2) principle-guided route planning, and (3) capability-based execution model selection. Additionally, we enhance the SPAgent's video quality evaluation capability, enabling it to autonomously assess and incorporate new video generation and editing models into its tool library without human intervention. Experimental results demonstrate that the SPAgent effectively coordinates models to generate or edit videos, highlighting its versatility and adaptability across various video tasks.
Presenting a new approach in security in inter-vehicle networks (VANET)
Venue: arXiv cs.CR 2024
Year: 2024

Nowadays, inter-vehicle networks are a viable communication scenario that greatly contributes to daily work, and its issues are gaining more and more attention every day. These days, space networks are growing and developing. There are numerous new uses for this new kind of network communication. One of the most significant daily programs in the world today is road traffic. For human growth, passenger and freight transportation is essential. Thus, fresh advancements in the areas of improved safety features, environmentally friendly fuel, etc., are developed daily. In order to improve safety and regulate traffic, a new application program is used. However, because of their stringent security standards, these initiatives have an impact on traffic safety. Since driving is one of the things that necessitates traffic safety, this area needs to be made more secure. Providing trustworthy driving data is crucial to achieving this goal, aside from the automated portion of the operation. Drivers would greatly benefit from accurate weather descriptions or early warnings of potential dangers (such as traffic bottlenecks or accidents). Inter-vehicle networks, a novel form of information technology, are being developed for this reason. Keywords: inter-vehicle network, transportation and security
Locally-Focused Face Representation for Sketch-to-Image Generation Using Noise-Induced Refinement
Venue: arXiv cs.CV 2024
Year: 2024

This paper presents a novel deep-learning framework that significantly enhances the transformation of rudimentary face sketches into high-fidelity colour images. Employing a Convolutional Block Attention-based Auto-encoder Network (CA2N), our approach effectively captures and enhances critical facial features through a block attention mechanism within an encoder-decoder architecture. Subsequently, the framework utilises a noise-induced conditional Generative Adversarial Network (cGAN) process that allows the system to maintain high performance even on domains unseen during the training. These enhancements lead to considerable improvements in image realism and fidelity, with our model achieving superior performance metrics that outperform the best method by FID margin of 17, 23, and 38 on CelebAMask-HQ, CUHK, and CUFSF datasets; respectively. The model sets a new state-of-the-art in sketch-to-image generation, can generalize across sketch types, and offers a robust solution for applications such as criminal identification in law enforcement.
Connectivity Preserving Decentralized UAV Swarm Navigation in Obstacle-laden Environments without Explicit Communication
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents a novel control method for a group of UAVs in obstacle-laden environments while preserving sensing network connectivity without data transmission between the UAVs. By leveraging constraints rooted in control barrier functions (CBFs), the proposed method aims to overcome the limitations, such as oscillatory behaviors and frequent constraint violations, of the existing method based on artificial potential fields (APFs). More specifically, the proposed method first determines desired control inputs by considering CBF-based constraints rather than repulsive APFs. The desired inputs are then minimally modified by solving a numerical optimization problem with soft constraints. In addition to the optimization-based method, we present an approximate method without numerical optimization. The effectiveness of the proposed methods is evaluated by extensive simulations to compare the performance of the CBF-based methods with an APF-based approach. Experimental results using real quadrotors are also presented.
Distributed matrix multiplication with straggler tolerance over very small field
Venue: arXiv cs.IT 2024
Year: 2024

The problem of distributed matrix multiplication with straggler tolerance over finite fields is considered, focusing on field sizes for which previous solutions were not applicable (for instance, the field of two elements). We employ Reed-Muller-type codes for explicitly constructing the desired algorithms and study their parameters by translating the problem into a combinatorial problem involving sums of discrete convex sets. We generalize polynomial codes and matdot codes, discussing the impossibility of the latter being applicable for very small field sizes, while providing optimal solutions for some regimes of parameters in both cases.
A Simple and Fast Algorithm for Fair Cuts
Venue: arXiv cs.DS 2024
Year: 2024

We present a simple and faster algorithm for computing fair cuts on undirected graphs, a concept introduced in recent work of Li et al. (SODA 2023). Informally, for any parameter $ε>0$, a $(1+ε)$-fair $(s,t)$-cut is an $(s,t)$-cut such that there exists an $(s,t)$-flow that uses $1/(1+ε)$ fraction of the capacity of every edge in the cut. Our algorithm computes a $(1+ε)$-fair cut in $\tilde O(m/ε)$ time, improving on the $\tilde O(m/ε^3)$ time algorithm of Li et al. and matching the $\tilde O(m/ε)$ time algorithm of Sherman (STOC 2017) for standard $(1+ε)$-approximate min-cut. Our main idea is to run Sherman's approximate max-flow/min-cut algorithm iteratively on a (directed) residual graph. While Sherman's algorithm is originally stated for undirected graphs, we show that it provides guarantees for directed graphs that are good enough for our purposes.