Orionnnn/hub-stats huggingface dataset:
ShortCircuit: AlphaZero-Driven Circuit Design
Authors: name: Dimitrios Tsaras, name: Antoine Grosnit, name: Lei Chen, name: Zhiyao Xie, name: Haitham Bou-Ammar, name: Mingxuan Yuan
Published: 2024-08-19 10:03:14

Chip design relies heavily on generating Boolean circuits, such as
AND-Inverter Graphs (AIGs), from functional descriptions like truth tables.
While recent advances in deep learning have aimed to accelerate circuit design,
these efforts have mostly focused on tasks other than synthesis, and
traditional heuristic methods have plateaued. In this paper, we introduce
ShortCircuit, a novel transformer-based architecture that leverages the
structural properties of AIGs and performs efficient space exploration.
Contrary to prior approaches attempting end-to-end generation of logic circuits
using deep networks, ShortCircuit employs a two-phase process combining
supervised with reinforcement learning to enhance generalization to unseen
truth tables. We also propose an AlphaZero variant to handle the double
exponentially large state space and the sparsity of the rewards, enabling the
discovery of near-optimal designs. To evaluate the generative performance of
our trained model , we extract 500 truth tables from a benchmark set of 20
real-world circuits. ShortCircuit successfully generates AIGs for 84.6% of the
8-input test truth tables, and outperforms the state-of-the-art logic synthesis
tool, ABC, by 14.61% in terms of circuits size.
SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse
  Views
Authors: name: Chao Xu, name: Ang Li, name: Linghao Chen, name: Yulin Liu, name: Ruoxi Shi, name: Hao Su, name: Minghua Liu
Published: 2024-08-19 17:53:10

Open-world 3D generation has recently attracted considerable attention. While
many single-image-to-3D methods have yielded visually appealing outcomes, they
often lack sufficient controllability and tend to produce hallucinated regions
that may not align with users' expectations. In this paper, we explore an
important scenario in which the input consists of one or a few unposed 2D
images of a single object, with little or no overlap. We propose a novel
method, SpaRP, to reconstruct a 3D textured mesh and estimate the relative
camera poses for these sparse-view images. SpaRP distills knowledge from 2D
diffusion models and finetunes them to implicitly deduce the 3D spatial
relationships between the sparse views. The diffusion model is trained to
jointly predict surrogate representations for camera poses and multi-view
images of the object under known poses, integrating all information from the
input sparse views. These predictions are then leveraged to accomplish 3D
reconstruction and pose estimation, and the reconstructed 3D model can be used
to further refine the camera poses of input views. Through extensive
experiments on three datasets, we demonstrate that our method not only
significantly outperforms baseline methods in terms of 3D reconstruction
quality and pose prediction accuracy but also exhibits strong efficiency. It
requires only about 20 seconds to produce a textured mesh and camera poses for
the input views. Project page: https://chaoxu.xyz/sparp.
TraDiffusion: Trajectory-Based Training-Free Image Generation
Authors: name: Mingrui Wu, name: Oucheng Huang, name: Jiayi Ji, name: Jiale Li, name: Xinyue Cai, name: Huafeng Kuang, name: Jianzhuang Liu, name: Xiaoshuai Sun, name: Rongrong Ji
Published: 2024-08-19 07:01:43

In this work, we propose a training-free, trajectory-based controllable T2I
approach, termed TraDiffusion. This novel method allows users to effortlessly
guide image generation via mouse trajectories. To achieve precise control, we
design a distance awareness energy function to effectively guide latent
variables, ensuring that the focus of generation is within the areas defined by
the trajectory. The energy function encompasses a control function to draw the
generation closer to the specified trajectory and a movement function to
diminish activity in areas distant from the trajectory. Through extensive
experiments and qualitative assessments on the COCO dataset, the results reveal
that TraDiffusion facilitates simpler, more natural image control. Moreover, it
showcases the ability to manipulate salient regions, attributes, and
relationships within the generated images, alongside visual input based on
arbitrary or enhanced trajectories.
Photorealistic Object Insertion with Diffusion-Guided Inverse Rendering
Authors: name: Ruofan Liang, name: Zan Gojcic, name: Merlin Nimier-David, name: David Acuna, name: Nandita Vijaykumar, name: Sanja Fidler, name: Zian Wang
Published: 2024-08-19 05:15:45

The correct insertion of virtual objects in images of real-world scenes
requires a deep understanding of the scene's lighting, geometry and materials,
as well as the image formation process. While recent large-scale diffusion
models have shown strong generative and inpainting capabilities, we find that
current models do not sufficiently "understand" the scene shown in a single
picture to generate consistent lighting effects (shadows, bright reflections,
etc.) while preserving the identity and details of the composited object. We
propose using a personalized large diffusion model as guidance to a physically
based inverse rendering process. Our method recovers scene lighting and
tone-mapping parameters, allowing the photorealistic composition of arbitrary
virtual objects in single frames or videos of indoor or outdoor scenes. Our
physically based pipeline further enables automatic materials and tone-mapping
refinement.
Segment Anything with Multiple Modalities
Authors: name: Aoran Xiao, name: Weihao Xuan, name: Heli Qi, name: Yun Xing, name: Naoto Yokoya, name: Shijian Lu
Published: 2024-08-17 03:45:40

Robust and accurate segmentation of scenes has become one core functionality
in various visual recognition and navigation tasks. This has inspired the
recent development of Segment Anything Model (SAM), a foundation model for
general mask segmentation. However, SAM is largely tailored for single-modal
RGB images, limiting its applicability to multi-modal data captured with
widely-adopted sensor suites, such as LiDAR plus RGB, depth plus RGB, thermal
plus RGB, etc. We develop MM-SAM, an extension and expansion of SAM that
supports cross-modal and multi-modal processing for robust and enhanced
segmentation with different sensor suites. MM-SAM features two key designs,
namely, unsupervised cross-modal transfer and weakly-supervised multi-modal
fusion, enabling label-efficient and parameter-efficient adaptation toward
various sensor modalities. It addresses three main challenges: 1) adaptation
toward diverse non-RGB sensors for single-modal processing, 2) synergistic
processing of multi-modal data via sensor fusion, and 3) mask-free training for
different downstream tasks. Extensive experiments show that MM-SAM consistently
outperforms SAM by large margins, demonstrating its effectiveness and
robustness across various sensors and data modalities.
Authorship Attribution in the Era of LLMs: Problems, Methodologies, and
  Challenges
Authors: name: Baixiang Huang, name: Canyu Chen, name: Kai Shu
Published: 2024-08-16 17:58:49

Accurate attribution of authorship is crucial for maintaining the integrity
of digital content, improving forensic investigations, and mitigating the risks
of misinformation and plagiarism. Addressing the imperative need for proper
authorship attribution is essential to uphold the credibility and
accountability of authentic authorship. The rapid advancements of Large
Language Models (LLMs) have blurred the lines between human and machine
authorship, posing significant challenges for traditional methods. We presents
a comprehensive literature review that examines the latest research on
authorship attribution in the era of LLMs. This survey systematically explores
the landscape of this field by categorizing four representative problems: (1)
Human-written Text Attribution; (2) LLM-generated Text Detection; (3)
LLM-generated Text Attribution; and (4) Human-LLM Co-authored Text Attribution.
We also discuss the challenges related to ensuring the generalization and
explainability of authorship attribution methods. Generalization requires the
ability to generalize across various domains, while explainability emphasizes
providing transparent and understandable insights into the decisions made by
these models. By evaluating the strengths and limitations of existing methods
and benchmarks, we identify key open problems and future research directions in
this field. This literature review serves a roadmap for researchers and
practitioners interested in understanding the state of the art in this rapidly
evolving field. Additional resources and a curated list of papers are available
and regularly updated at https://llm-authorship.github.io
Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risk
  of Language Models
Authors: name: Andy K. Zhang, name: Neil Perry, name: Riya Dulepet, name: Eliot Jones, name: Justin W. Lin, name: Joey Ji, name: Celeste Menders, name: Gashon Hussein, name: Samantha Liu, name: Donovan Jasper, name: Pura Peetathawatchai, name: Ari Glenn, name: Vikram Sivashankar, name: Daniel Zamoshchin, name: Leo Glikbarg, name: Derek Askaryar, name: Mike Yang, name: Teddy Zhang, name: Rishi Alluri, name: Nathan Tran, name: Rinnara Sangpisit, name: Polycarpos Yiorkadjis, name: Kenny Osele, name: Gautham Raghupathi, name: Dan Boneh, name: Daniel E. Ho, name: Percy Liang
Published: 2024-08-15 17:23:10

Language Model (LM) agents for cybersecurity that are capable of autonomously
identifying vulnerabilities and executing exploits have the potential to cause
real-world impact. Policymakers, model providers, and other researchers in the
AI and cybersecurity communities are interested in quantifying the capabilities
of such agents to help mitigate cyberrisk and investigate opportunities for
penetration testing. Toward that end, we introduce Cybench, a framework for
specifying cybersecurity tasks and evaluating agents on those tasks. We include
40 professional-level Capture the Flag (CTF) tasks from 4 distinct CTF
competitions, chosen to be recent, meaningful, and spanning a wide range of
difficulties. Each task includes its own description, starter files, and is
initialized in an environment where an agent can execute bash commands and
observe outputs. Since many tasks are beyond the capabilities of existing LM
agents, we introduce subtasks, which break down a task into intermediary steps
for more gradated evaluation; we add subtasks for 17 of the 40 tasks. To
evaluate agent capabilities, we construct a cybersecurity agent and evaluate 7
models: GPT-4o, Claude 3 Opus, Claude 3.5 Sonnet, Mixtral 8x22b Instruct,
Gemini 1.5 Pro, Llama 3 70B Chat, and Llama 3.1 405B Instruct. Without
guidance, we find that agents are able to solve only the easiest complete tasks
that took human teams up to 11 minutes to solve, with Claude 3.5 Sonnet and
GPT-4o having the highest success rates. Finally, subtasks provide more signal
for measuring performance compared to unguided runs, with models achieving a
3.2\% higher success rate on complete tasks with subtask-guidance than without
subtask-guidance. All code and data are publicly available at
https://cybench.github.io
LongVILA: Scaling Long-Context Visual Language Models for Long Videos
Authors: name: Fuzhao Xue, name: Yukang Chen, name: Dacheng Li, name: Qinghao Hu, name: Ligeng Zhu, name: Xiuyu Li, name: Yunhao Fang, name: Haotian Tang, name: Shang Yang, name: Zhijian Liu, name: Ethan He, name: Hongxu Yin, name: Pavlo Molchanov, name: Jan Kautz, name: Linxi Fan, name: Yuke Zhu, name: Yao Lu, name: Song Han
Published: 2024-08-19 17:48:08

Long-context capability is critical for multi-modal foundation models. We
introduce LongVILA, a full-stack solution for long-context vision-language
models, including system, model training, and dataset development. On the
system side, we introduce the first Multi-Modal Sequence Parallelism (MM-SP)
system that enables long-context training and inference, enabling 2M context
length training on 256 GPUs. MM-SP is also efficient, being 2.1x - 5.7x faster
than Ring-Style Sequence Parallelism and 1.1x - 1.4x faster than Megatron-LM in
text-only settings. Moreover, it seamlessly integrates with Hugging Face
Transformers. For model training, we propose a five-stage pipeline comprising
alignment, pre-training, context extension, and long-short joint supervised
fine-tuning. Regarding datasets, we meticulously construct large-scale visual
language pre-training datasets and long video instruction-following datasets to
support our multi-stage training process. The full-stack solution extends the
feasible frame number of VILA by a factor of 128 (from 8 to 1024 frames) and
improves long video captioning score from 2.00 to 3.26 (1.6x), achieving 99.5%
accuracy in 1400-frames video (274k context length) needle in a haystack.
LongVILA-8B also demonstrates a consistent improvement in performance on long
videos within the VideoMME benchmark as the video frames increase.
Factorized-Dreamer: Training A High-Quality Video Generator with Limited
  and Low-Quality Data
Authors: name: Tao Yang, name: Yangming Shi, name: Yunwen Huang, name: Feng Chen, name: Yin Zheng, name: Lei Zhang
Published: 2024-08-19 16:08:00

Text-to-video (T2V) generation has gained significant attention due to its
wide applications to video generation, editing, enhancement and translation,
\etc. However, high-quality (HQ) video synthesis is extremely challenging
because of the diverse and complex motions existed in real world. Most existing
works struggle to address this problem by collecting large-scale HQ videos,
which are inaccessible to the community. In this work, we show that publicly
available limited and low-quality (LQ) data are sufficient to train a HQ video
generator without recaptioning or finetuning. We factorize the whole T2V
generation process into two steps: generating an image conditioned on a highly
descriptive caption, and synthesizing the video conditioned on the generated
image and a concise caption of motion details. Specifically, we present
Factorized-Dreamer, a factorized spatiotemporal framework with several
critical designs for T2V generation, including an adapter to combine text and
image embeddings, a pixel-aware cross attention module to capture pixel-level
image information, a T5 text encoder to better understand motion description,
and a PredictNet to supervise optical flows. We further present a noise
schedule, which plays a key role in ensuring the quality and stability of video
generation. Our model lowers the requirements in detailed captions and HQ
videos, and can be directly trained on limited LQ datasets with noisy and brief
captions such as WebVid-10M, largely alleviating the cost to collect
large-scale HQ video-text pairs. Extensive experiments in a variety of T2V and
image-to-video generation tasks demonstrate the effectiveness of our proposed
Factorized-Dreamer. Our source codes are available at
https://github.com/yangxy/Factorized-Dreamer/.
MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
  Model
Authors: name: Minghua Liu, name: Chong Zeng, name: Xinyue Wei, name: Ruoxi Shi, name: Linghao Chen, name: Chao Xu, name: Mengqi Zhang, name: Zhaoning Wang, name: Xiaoshuai Zhang, name: Isabella Liu, name: Hongzhi Wu, name: Hao Su
Published: 2024-08-19 17:55:17

Open-world 3D reconstruction models have recently garnered significant
attention. However, without sufficient 3D inductive bias, existing methods
typically entail expensive training costs and struggle to extract high-quality
3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
model that explicitly leverages 3D native structure, input guidance, and
training supervision. Specifically, instead of using a triplane representation,
we store features in 3D sparse voxels and combine transformers with 3D
convolutions to leverage an explicit 3D structure and projective bias. In
addition to sparse-view RGB input, we require the network to take input and
generate corresponding normal maps. The input normal maps can be predicted by
2D diffusion models, significantly aiding in the guidance and refinement of the
geometry's learning. Moreover, by combining Signed Distance Function (SDF)
supervision with surface rendering, we directly learn to generate high-quality
meshes without the need for complex multi-stage training processes. By
incorporating these explicit 3D biases, MeshFormer can be trained efficiently
and deliver high-quality textured meshes with fine-grained geometric details.
It can also be integrated with 2D diffusion models to enable fast
single-image-to-3D and text-to-3D tasks. Project page:
https://meshformer3d.github.io
NeuFlow v2: High-Efficiency Optical Flow Estimation on Edge Devices
Authors: name: Zhiyong Zhang, name: Aniket Gupta, name: Huaizu Jiang, name: Hanumant Singh
Published: 2024-08-19 17:13:34

Real-time high-accuracy optical flow estimation is crucial for various
real-world applications. While recent learning-based optical flow methods have
achieved high accuracy, they often come with significant computational costs.
In this paper, we propose a highly efficient optical flow method that balances
high accuracy with reduced computational demands. Building upon NeuFlow v1, we
introduce new components including a much more light-weight backbone and a fast
refinement module. Both these modules help in keeping the computational demands
light while providing close to state of the art accuracy. Compares to other
state of the art methods, our model achieves a 10x-70x speedup while
maintaining comparable performance on both synthetic and real-world data. It is
capable of running at over 20 FPS on 512x384 resolution images on a Jetson Orin
Nano. The full training and evaluation code is available at
https://github.com/neufieldrobotics/NeuFlow_v2.
Fine-tuning Large Language Models with Human-inspired Learning
  Strategies in Medical Question Answering
Authors: name: Yushi Yang, name: Andrew M. Bean, name: Robert McCraith, name: Adam Mahdi
Published: 2024-08-15 02:22:48

Training Large Language Models (LLMs) incurs substantial data-related costs,
motivating the development of data-efficient training methods through optimised
data ordering and selection. Human-inspired learning strategies, such as
curriculum learning, offer possibilities for efficient training by organising
data according to common human learning practices. Despite evidence that
fine-tuning with curriculum learning improves the performance of LLMs for
natural language understanding tasks, its effectiveness is typically assessed
using a single model. In this work, we extend previous research by evaluating
both curriculum-based and non-curriculum-based learning strategies across
multiple LLMs, using human-defined and automated data labels for medical
question answering. Our results indicate a moderate impact of using
human-inspired learning strategies for fine-tuning LLMs, with maximum accuracy
gains of 1.77% per model and 1.81% per dataset. Crucially, we demonstrate that
the effectiveness of these strategies varies significantly across different
model-dataset combinations, emphasising that the benefits of a specific
human-inspired strategy for fine-tuning LLMs do not generalise. Additionally,
we find evidence that curriculum learning using LLM-defined question difficulty
outperforms human-defined difficulty, highlighting the potential of using
model-generated measures for optimal curriculum design.
Surgical SAM 2: Real-time Segment Anything in Surgical Video by
  Efficient Frame Pruning
Authors: name: Haofeng Liu, name: Erli Zhang, name: Junde Wu, name: Mingxuan Hong, name: Yueming Jin
Published: 2024-08-15 04:59:12

Surgical video segmentation is a critical task in computer-assisted surgery
and is vital for enhancing surgical quality and patient outcomes. Recently, the
Segment Anything Model 2 (SAM2) framework has shown superior advancements in
image and video segmentation. However, SAM2 struggles with efficiency due to
the high computational demands of processing high-resolution images and complex
and long-range temporal dynamics in surgical videos. To address these
challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to
utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate
real-time surgical video segmentation. The EFP mechanism dynamically manages
the memory bank by selectively retaining only the most informative frames,
reducing memory usage and computational cost while maintaining high
segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2
significantly improves both efficiency and segmentation accuracy compared to
the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3times FPS compared with
SAM2, while also delivering state-of-the-art performance after fine-tuning with
lower-resolution data. These advancements establish SurgSAM-2 as a leading
model for surgical video analysis, making real-time surgical video segmentation
in resource-constrained environments a feasible reality.
xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
Authors: name: Le Xue, name: Manli Shu, name: Anas Awadalla, name: Jun Wang, name: An Yan, name: Senthil Purushwalkam, name: Honglu Zhou, name: Viraj Prabhu, name: Yutong Dai, name: Michael S Ryoo, name: Shrikant Kendre, name: Jieyu Zhang, name: Can Qin, name: Shu Zhang, name: Chia-Chih Chen, name: Ning Yu, name: Juntao Tan, name: Tulika Manoj Awalgaonkar, name: Shelby Heinecke, name: Huan Wang, name: Yejin Choi, name: Ludwig Schmidt, name: Zeyuan Chen, name: Silvio Savarese, name: Juan Carlos Niebles, name: Caiming Xiong, name: Ran Xu
Published: 2024-08-16 17:57:01

This report introduces xGen-MM (also known as BLIP-3), a framework for
developing Large Multimodal Models (LMMs). The framework comprises meticulously
curated datasets, a training recipe, model architectures, and a resulting suite
of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen
initiative on foundation AI models. Our models undergo rigorous evaluation
across a range of tasks, including both single and multi-image benchmarks. Our
pre-trained base model exhibits strong in-context learning capabilities and the
instruction-tuned model demonstrates competitive performance among open-source
LMMs with similar model sizes. In addition, we introduce a safety-tuned model
with DPO, aiming to mitigate harmful behaviors such as hallucinations and
improve safety. We open-source our models, curated large-scale datasets, and
our fine-tuning codebase to facilitate further advancements in LMM research.
Associated resources will be available on our project page above.