Orionnnn/hub-stats huggingface dataset:
TrackGo: A Flexible and Efficient Method for Controllable Video
  Generation
Authors: name: Haitao Zhou, name: Chuang Wang, name: Rui Nie, name: Jinxiao Lin, name: Dongdong Yu, name: Qian Yu, name: Changhu Wang
Published: 2024-08-21 09:42:04

Recent years have seen substantial progress in diffusion-based controllable
video generation. However, achieving precise control in complex scenarios,
including fine-grained object parts, sophisticated motion trajectories, and
coherent background movement, remains a challenge. In this paper, we introduce
TrackGo, a novel approach that leverages free-form masks and arrows for
conditional video generation. This method offers users with a flexible and
precise mechanism for manipulating video content. We also propose the
TrackAdapter for control implementation, an efficient and lightweight adapter
designed to be seamlessly integrated into the temporal self-attention layers of
a pretrained video generation model. This design leverages our observation that
the attention map of these layers can accurately activate regions corresponding
to motion in videos. Our experimental results demonstrate that our new
approach, enhanced by the TrackAdapter, achieves state-of-the-art performance
on key metrics such as FVD, FID, and ObjMC scores. The project page of TrackGo
can be found at: https://zhtjtcz.github.io/TrackGo-Page/
NeCo: Improving DINOv2's spatial representations in 19 GPU hours with
  Patch Neighbor Consistency
Authors: name: Valentinos Pariza, name: Mohammadreza Salehi, name: Gertjan Burghouts, name: Francesco Locatello, name: Yuki M. Asano
Published: 2024-08-20 17:58:59

We propose sorting patch representations across views as a novel
self-supervised learning signal to improve pretrained representations. To this
end, we introduce NeCo: Patch Neighbor Consistency, a novel training loss that
enforces patch-level nearest neighbor consistency across a student and teacher
model, relative to reference batches. Our method leverages a differentiable
sorting method applied on top of pretrained representations, such as
DINOv2-registers to bootstrap the learning signal and further improve upon
them. This dense post-pretraining leads to superior performance across various
models and datasets, despite requiring only 19 hours on a single GPU. We
demonstrate that this method generates high-quality dense feature encoders and
establish several new state-of-the-art results: +5.5% and + 6% for
non-parametric in-context semantic segmentation on ADE20k and Pascal VOC, and
+7.2% and +5.7% for linear segmentation evaluations on COCO-Things and -Stuff.
PhysBERT: A Text Embedding Model for Physics Scientific Literature
Authors: name: Thorsten Hellert, name: João Montenegro, name: Andrea Pollastro
Published: 2024-08-18 19:18:12

The specialized language and complex concepts in physics pose significant
challenges for information extraction through Natural Language Processing
(NLP). Central to effective NLP applications is the text embedding model, which
converts text into dense vector representations for efficient information
retrieval and semantic analysis. In this work, we introduce PhysBERT, the first
physics-specific text embedding model. Pre-trained on a curated corpus of 1.2
million arXiv physics papers and fine-tuned with supervised data, PhysBERT
outperforms leading general-purpose models on physics-specific tasks including
the effectiveness in fine-tuning for specific physics subdomains.
Recent Surge in Public Interest in Transportation: Sentiment Analysis of
  Baidu Apollo Go Using Weibo Data
Authors: name: Shiqi Wang, name: Zhouye Zhao, name: Yuhang Xie, name: Mingchuan Ma, name: Zirui Chen, name: Zeyu Wang, name: Bohao Su, name: Wenrui Xu, name: Tianyi Li
Published: 2024-08-19 15:29:56

Urban mobility and transportation systems have been profoundly transformed by
the advancement of autonomous vehicle technologies. Baidu Apollo Go, a pioneer
robotaxi service from the Chinese tech giant Baidu, has recently been widely
deployed in major cities like Beijing and Wuhan, sparking increased
conversation and offering a glimpse into the future of urban mobility.
  This study investigates public attitudes towards Apollo Go across China using
Sentiment Analysis with a hybrid BERT model on 36,096 Weibo posts from January
to July 2024. The analysis shows that 89.56\% of posts related to Apollo Go are
clustered in July. From January to July, public sentiment was mostly positive,
but negative comments began to rise after it became a hot topic on July 21.
  Spatial analysis indicates a strong correlation between provinces with high
discussion intensity and those where Apollo Go operates. Initially, Hubei and
Guangdong dominated online posting volume, but by July, Guangdong, Beijing, and
international regions had overtaken Hubei. Attitudes varied significantly among
provinces, with Xinjiang and Qinghai showing optimism and Tibet and Gansu
expressing concerns about the impact on traditional taxi services.
  Sentiment analysis revealed that positive comments focused on technology
applications and personal experiences, while negative comments centered on job
displacement and safety concerns. In summary, this study highlights the
divergence in public perceptions of autonomous ride-hailing services, providing
valuable insights for planners, policymakers, and service providers. The model
is published on Hugging Face at
https://huggingface.co/wsqstar/bert-finetuned-weibo-luobokuaipao and the
repository on GitHub at https://github.com/GIStudio/trb2024.
Ferret: Faster and Effective Automated Red Teaming with Reward-Based
  Scoring Technique
Authors: name: Tej Deep Pala, name: Vernon Y. H. Toh, name: Rishabh Bhardwaj, name: Soujanya Poria
Published: 2024-08-20 09:58:01

In today's era, where large language models (LLMs) are integrated into
numerous real-world applications, ensuring their safety and robustness is
crucial for responsible AI usage. Automated red-teaming methods play a key role
in this process by generating adversarial attacks to identify and mitigate
potential vulnerabilities in these models. However, existing methods often
struggle with slow performance, limited categorical diversity, and high
resource demands. While Rainbow Teaming, a recent approach, addresses the
diversity challenge by framing adversarial prompt generation as a
quality-diversity search, it remains slow and requires a large fine-tuned
mutator for optimal performance. To overcome these limitations, we propose
Ferret, a novel approach that builds upon Rainbow Teaming by generating
multiple adversarial prompt mutations per iteration and using a scoring
function to rank and select the most effective adversarial prompt. We explore
various scoring functions, including reward models, Llama Guard, and
LLM-as-a-judge, to rank adversarial mutations based on their potential harm to
improve the efficiency of the search for harmful mutations. Our results
demonstrate that Ferret, utilizing a reward model as a scoring function,
improves the overall attack success rate (ASR) to 95%, which is 46% higher than
Rainbow Teaming. Additionally, Ferret reduces the time needed to achieve a 90%
ASR by 15.2% compared to the baseline and generates adversarial prompts that
are transferable i.e. effective on other LLMs of larger size. Our codes are
available at https://github.com/declare-lab/ferret.
The Brittleness of AI-Generated Image Watermarking Techniques: Examining
  Their Robustness Against Visual Paraphrasing Attacks
Authors: name: Niyar R Barman, name: Krish Sharma, name: Ashhar Aziz, name: Shashwat Bajpai, name: Shwetangshu Biswas, name: Vasu Sharma, name: Vinija Jain, name: Aman Chadha, name: Amit Sheth, name: Amitava Das
Published: 2024-08-19 22:58:30

The rapid advancement of text-to-image generation systems, exemplified by
models like Stable Diffusion, Midjourney, Imagen, and DALL-E, has heightened
concerns about their potential misuse. In response, companies like Meta and
Google have intensified their efforts to implement watermarking techniques on
AI-generated images to curb the circulation of potentially misleading visuals.
However, in this paper, we argue that current image watermarking methods are
fragile and susceptible to being circumvented through visual paraphrase
attacks. The proposed visual paraphraser operates in two steps. First, it
generates a caption for the given image using KOSMOS-2, one of the latest
state-of-the-art image captioning systems. Second, it passes both the original
image and the generated caption to an image-to-image diffusion system. During
the denoising step of the diffusion pipeline, the system generates a visually
similar image that is guided by the text caption. The resulting image is a
visual paraphrase and is free of any watermarks. Our empirical findings
demonstrate that visual paraphrase attacks can effectively remove watermarks
from images. This paper provides a critical assessment, empirically revealing
the vulnerability of existing watermarking techniques to visual paraphrase
attacks. While we do not propose solutions to this issue, this paper serves as
a call to action for the scientific community to prioritize the development of
more robust watermarking techniques. Our first-of-its-kind visual paraphrase
dataset and accompanying code are publicly available.
TableBench: A Comprehensive and Complex Benchmark for Table Question
  Answering
Authors: name: Xianjie Wu, name: Jian Yang, name: Linzheng Chai, name: Ge Zhang, name: Jiaheng Liu, name: Xinrun Du, name: Di Liang, name: Daixin Shu, name: Xianfu Cheng, name: Tianzhen Sun, name: Guanglin Niu, name: Tongliang Li, name: Zhoujun Li
Published: 2024-08-17 11:40:10

Recent advancements in Large Language Models (LLMs) have markedly enhanced
the interpretation and processing of tabular data, introducing previously
unimaginable capabilities. Despite these achievements, LLMs still encounter
significant challenges when applied in industrial scenarios, particularly due
to the increased complexity of reasoning required with real-world tabular data,
underscoring a notable disparity between academic benchmarks and practical
applications. To address this discrepancy, we conduct a detailed investigation
into the application of tabular data in industrial scenarios and propose a
comprehensive and complex benchmark TableBench, including 18 fields within four
major categories of table question answering (TableQA) capabilities.
Furthermore, we introduce TableLLM, trained on our meticulously constructed
training set TableInstruct, achieving comparable performance with GPT-3.5.
Massive experiments conducted on TableBench indicate that both open-source and
proprietary LLMs still have significant room for improvement to meet real-world
demands, where the most advanced model, GPT-4, achieves only a modest score
compared to humans.
Predicting Rewards Alongside Tokens: Non-disruptive Parameter Insertion
  for Efficient Inference Intervention in Large Language Model
Authors: name: Chenhan Yuan, name: Fei Huang, name: Ru Peng, name: Keming Lu, name: Bowen Yu, name: Chang Zhou, name: Jingren Zhou
Published: 2024-08-20 12:00:35

Transformer-based large language models (LLMs) exhibit limitations such as
generating unsafe responses, unreliable reasoning, etc. Existing inference
intervention approaches attempt to mitigate these issues by finetuning
additional models to produce calibration signals (such as rewards) that guide
the LLM's decoding process. However, this solution introduces substantial time
and space overhead due to the separate models required. This work proposes
Non-disruptive parameters insertion (Otter), inserting extra parameters into
the transformer architecture to predict calibration signals along with the
original LLM output. Otter offers state-of-the-art performance on multiple
demanding tasks while saving up to 86.5\% extra space and 98.5\% extra time.
Furthermore, Otter seamlessly integrates with existing inference engines,
requiring only a one-line code change, and the original model response remains
accessible after the parameter insertion. Our code is publicly available at
https://github.com/chenhan97/Otter
RP1M: A Large-Scale Motion Dataset for Piano Playing with Bi-Manual
  Dexterous Robot Hands
Authors: name: Yi Zhao, name: Le Chen, name: Jan Schneider, name: Quankai Gao, name: Juho Kannala, name: Bernhard Schölkopf, name: Joni Pajarinen, name: Dieter Büchler
Published: 2024-08-20 17:56:52

It has been a long-standing research goal to endow robot hands with
human-level dexterity. Bi-manual robot piano playing constitutes a task that
combines challenges from dynamic tasks, such as generating fast while precise
motions, with slower but contact-rich manipulation problems. Although
reinforcement learning based approaches have shown promising results in
single-task performance, these methods struggle in a multi-song setting. Our
work aims to close this gap and, thereby, enable imitation learning approaches
for robot piano playing at scale. To this end, we introduce the Robot Piano 1
Million (RP1M) dataset, containing bi-manual robot piano playing motion data of
more than one million trajectories. We formulate finger placements as an
optimal transport problem, thus, enabling automatic annotation of vast amounts
of unlabeled songs. Benchmarking existing imitation learning approaches shows
that such approaches reach state-of-the-art robot piano playing performance by
leveraging RP1M.
ShapeSplat: A Large-scale Dataset of Gaussian Splats and Their
  Self-Supervised Pretraining
Authors: name: Qi Ma, name: Yue Li, name: Bin Ren, name: Nicu Sebe, name: Ender Konukoglu, name: Theo Gevers, name: Luc Van Gool, name: Danda Pani Paudel
Published: 2024-08-20 14:49:14

3D Gaussian Splatting (3DGS) has become the de facto method of 3D
representation in many vision tasks. This calls for the 3D understanding
directly in this representation space. To facilitate the research in this
direction, we first build a large-scale dataset of 3DGS using the commonly used
ShapeNet and ModelNet datasets. Our dataset ShapeSplat consists of 65K objects
from 87 unique categories, whose labels are in accordance with the respective
datasets. The creation of this dataset utilized the compute equivalent of 2 GPU
years on a TITAN XP GPU.
  We utilize our dataset for unsupervised pretraining and supervised finetuning
for classification and segmentation tasks. To this end, we introduce
\textit{Gaussian-MAE}, which highlights the unique benefits of
representation learning from Gaussian parameters. Through exhaustive
experiments, we provide several valuable insights. In particular, we show that
(1) the distribution of the optimized GS centroids significantly differs from
the uniformly sampled point cloud (used for initialization) counterpart; (2)
this change in distribution results in degradation in classification but
improvement in segmentation tasks when using only the centroids; (3) to
leverage additional Gaussian parameters, we propose Gaussian feature grouping
in a normalized feature space, along with splats pooling layer, offering a
tailored solution to effectively group and embed similar Gaussians, which leads
to notable improvement in finetuning tasks.
MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context
  Generation with Speculative Decoding
Authors: name: Jian Chen, name: Vashisth Tiwari, name: Ranajoy Sadhukhan, name: Zhuoming Chen, name: Jinyuan Shi, name: Ian En-Hsu Yen, name: Beidi Chen
Published: 2024-08-20 17:57:31

Large Language Models (LLMs) have become more prevalent in long-context
applications such as interactive chatbots, document analysis, and agent
workflows, but it is challenging to serve long-context requests with low
latency and high throughput. Speculative decoding (SD) is a widely used
technique to reduce latency without sacrificing performance but the
conventional wisdom suggests that its efficacy is limited to small batch sizes.
In MagicDec, we show that surprisingly SD can achieve speedup even for a high
throughput inference regime for moderate to long sequences. More interestingly,
an intelligent drafting strategy can achieve better speedup with increasing
batch size based on our rigorous analysis. MagicDec first identifies the
bottleneck shifts with increasing batch size and sequence length, and uses
these insights to deploy speculative decoding more effectively for high
throughput inference. Then, it leverages draft models with sparse KV cache to
address the KV bottleneck that scales with both sequence length and batch size.
To Code, or Not To Code? Exploring Impact of Code in Pre-training
Authors: name: Viraat Aryabumi, name: Yixuan Su, name: Raymond Ma, name: Adrien Morisot, name: Ivan Zhang, name: Acyr Locatelli, name: Marzieh Fadaee, name: Ahmet Üstün, name: Sara Hooker
Published: 2024-08-20 14:58:13

Including code in the pre-training data mixture, even for models not
specifically designed for code, has become a common practice in LLMs
pre-training. While there has been anecdotal consensus among practitioners that
code data plays a vital role in general LLMs' performance, there is only
limited work analyzing the precise impact of code on non-code tasks. In this
work, we systematically investigate the impact of code data on general
performance. We ask "what is the impact of code data used in pre-training on a
large variety of downstream tasks beyond code generation". We conduct extensive
ablations and evaluate across a broad range of natural language reasoning
tasks, world knowledge tasks, code benchmarks, and LLM-as-a-judge win-rates for
models with sizes ranging from 470M to 2.8B parameters. Across settings, we
find a consistent results that code is a critical building block for
generalization far beyond coding tasks and improvements to code quality have an
outsized impact across all tasks. In particular, compared to text-only
pre-training, the addition of code results in up to relative increase of 8.2%
in natural language (NL) reasoning, 4.2% in world knowledge, 6.6% improvement
in generative win-rates, and a 12x boost in code performance respectively. Our
work suggests investments in code quality and preserving code during
pre-training have positive impacts.
Transfusion: Predict the Next Token and Diffuse Images with One
  Multi-Modal Model
Authors: name: Chunting Zhou, name: Lili Yu, name: Arun Babu, name: Kushal Tirumala, name: Michihiro Yasunaga, name: Leonid Shamis, name: Jacob Kahn, name: Xuezhe Ma, name: Luke Zettlemoyer, name: Omer Levy
Published: 2024-08-20 17:48:20

We introduce Transfusion, a recipe for training a multi-modal model over
discrete and continuous data. Transfusion combines the language modeling loss
function (next token prediction) with diffusion to train a single transformer
over mixed-modality sequences. We pretrain multiple Transfusion models up to 7B
parameters from scratch on a mixture of text and image data, establishing
scaling laws with respect to a variety of uni- and cross-modal benchmarks. Our
experiments show that Transfusion scales significantly better than quantizing
images and training a language model over discrete image tokens. By introducing
modality-specific encoding and decoding layers, we can further improve the
performance of Transfusion models, and even compress each image to just 16
patches. We further demonstrate that scaling our Transfusion recipe to 7B
parameters and 2T multi-modal tokens produces a model that can generate images
and text on a par with similar scale diffusion models and language models,
reaping the benefits of both worlds.
Audio Match Cutting: Finding and Creating Matching Audio Transitions in
  Movies and Videos
Authors: name: Dennis Fedorishin, name: Lie Lu, name: Srirangaraj Setlur, name: Venu Govindaraju
Published: 2024-08-20 16:46:54

A "match cut" is a common video editing technique where a pair of shots that
have a similar composition transition fluidly from one to another. Although
match cuts are often visual, certain match cuts involve the fluid transition of
audio, where sounds from different sources merge into one indistinguishable
transition between two shots. In this paper, we explore the ability to
automatically find and create "audio match cuts" within videos and movies. We
create a self-supervised audio representation for audio match cutting and
develop a coarse-to-fine audio match pipeline that recommends matching shots
and creates the blended audio. We further annotate a dataset for the proposed
audio match cut task and compare the ability of multiple audio representations
to find audio match cut candidates. Finally, we evaluate multiple methods to
blend two matching audio candidates with the goal of creating a smooth
transition. Project page and examples are available at:
https://denfed.github.io/audiomatchcut/
MambaEVT: Event Stream based Visual Object Tracking using State Space
  Model
Authors: name: Xiao Wang, name: Chao wang, name: Shiao Wang, name: Xixi Wang, name: Zhicheng Zhao, name: Lin Zhu, name: Bo Jiang
Published: 2024-08-20 02:01:17

Event camera-based visual tracking has drawn more and more attention in
recent years due to the unique imaging principle and advantages of low energy
consumption, high dynamic range, and dense temporal resolution. Current
event-based tracking algorithms are gradually hitting their performance
bottlenecks, due to the utilization of vision Transformer and the static
template for target object localization. In this paper, we propose a novel
Mamba-based visual tracking framework that adopts the state space model with
linear complexity as a backbone network. The search regions and target template
are fed into the vision Mamba network for simultaneous feature extraction and
interaction. The output tokens of search regions will be fed into the tracking
head for target localization. More importantly, we consider introducing a
dynamic template update strategy into the tracking framework using the Memory
Mamba network. By considering the diversity of samples in the target template
library and making appropriate adjustments to the template memory module, a
more effective dynamic template can be integrated. The effective combination of
dynamic and static templates allows our Mamba-based tracking algorithm to
achieve a good balance between accuracy and computational cost on multiple
large-scale datasets, including EventVOT, VisEvent, and FE240hz. The source
code will be released on https://github.com/Event-AHU/MambaEVT
MegaFusion: Extend Diffusion Models towards Higher-resolution Image
  Generation without Further Tuning
Authors: name: Haoning Wu, name: Shaocheng Shen, name: Qiang Hu, name: Xiaoyun Zhang, name: Ya Zhang, name: Yanfeng Wang
Published: 2024-08-20 16:53:34

Diffusion models have emerged as frontrunners in text-to-image generation for
their impressive capabilities. Nonetheless, their fixed image resolution during
training often leads to challenges in high-resolution image generation, such as
semantic inaccuracies and object replication. This paper introduces MegaFusion,
a novel approach that extends existing diffusion-based text-to-image generation
models towards efficient higher-resolution generation without additional
fine-tuning or extra adaptation. Specifically, we employ an innovative truncate
and relay strategy to bridge the denoising processes across different
resolutions, allowing for high-resolution image generation in a coarse-to-fine
manner. Moreover, by integrating dilated convolutions and noise re-scheduling,
we further adapt the model's priors for higher resolution. The versatility and
efficacy of MegaFusion make it universally applicable to both latent-space and
pixel-space diffusion models, along with other derivative models. Extensive
experiments confirm that MegaFusion significantly boosts the capability of
existing models to produce images of megapixels and various aspect ratios,
while only requiring about 40% of the original computational cost.