Orionnnn/hub-stats huggingface dataset:
Non Verbis, Sed Rebus: Large Language Models are Weak Solvers of Italian
  Rebuses
Authors: name: Gabriele Sarti, name: Tommaso Caselli, name: Malvina Nissim, name: Arianna Bisazza
Published: 2024-08-01 14:14:15

Rebuses are puzzles requiring constrained multi-step reasoning to identify a
hidden phrase from a set of images and letters. In this work, we introduce a
large collection of verbalized rebuses for the Italian language and use it to
assess the rebus-solving capabilities of state-of-the-art large language
models. While general-purpose systems such as LLaMA-3 and GPT-4o perform poorly
on this task, ad-hoc fine-tuning seems to improve models' performance. However,
we find that performance gains from training are largely motivated by
memorization. Our results suggest that rebus solving remains a challenging test
bed to evaluate large language models' linguistic proficiency and sequential
instruction-following skills.
MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models
  for Integrated Capabilities
Authors: name: Weihao Yu, name: Zhengyuan Yang, name: Linfeng Ren, name: Linjie Li, name: Jianfeng Wang, name: Kevin Lin, name: Chung-Ching Lin, name: Zicheng Liu, name: Lijuan Wang, name: Xinchao Wang
Published: 2024-08-01 17:59:54

MM-Vet, with open-ended vision-language questions targeting at evaluating
integrated capabilities, has become one of the most popular benchmarks for
large multimodal model evaluation. MM-Vet assesses six core vision-language
(VL) capabilities: recognition, knowledge, spatial awareness, language
generation, OCR, and math. However, its question format is restricted to single
image-text pairs, lacking the interleaved image and text sequences prevalent in
real-world scenarios. To address this limitation, we introduce MM-Vet v2, which
includes a new VL capability called "image-text sequence understanding",
evaluating models' ability to process VL sequences. Furthermore, we maintain
the high quality of evaluation samples while further expanding the evaluation
set size. Using MM-Vet v2 to benchmark large multimodal models, we found that
Claude 3.5 Sonnet is the best model with a score of 71.8, slightly
outperforming GPT-4o which scored 71.0. Among open-weight models,
InternVL2-Llama3-76B leads with a score of 68.4.
OmniParser for Pure Vision Based GUI Agent
Authors: name: Yadong Lu, name: Jianwei Yang, name: Yelong Shen, name: Ahmed Awadallah
Published: 2024-08-01 00:00:43

The recent success of large vision language models shows great potential in
driving the agent system operating on user interfaces. However, we argue that
the power multimodal models like GPT-4V as a general agent on multiple
operating systems across different applications is largely underestimated due
to the lack of a robust screen parsing technique capable of: 1) reliably
identifying interactable icons within the user interface, and 2) understanding
the semantics of various elements in a screenshot and accurately associate the
intended action with the corresponding region on the screen. To fill these
gaps, we introduce OmniParser, a comprehensive method for parsing user
interface screenshots into structured elements, which significantly enhances
the ability of GPT-4V to generate actions that can be accurately grounded in
the corresponding regions of the interface. We first curated an interactable
icon detection dataset using popular webpages and an icon description dataset.
These datasets were utilized to fine-tune specialized models: a detection model
to parse interactable regions on the screen and a caption model to extract the
functional semantics of the detected elements. OmniParser
significantly improves GPT-4V's performance on ScreenSpot benchmark. And on
Mind2Web and AITW benchmark, OmniParser with screenshot only input
outperforms the GPT-4V baselines requiring additional information outside of
screenshot.
UniTalker: Scaling up Audio-Driven 3D Facial Animation through A Unified
  Model
Authors: name: Xiangyu Fan, name: Jiaqi Li, name: Zhiqian Lin, name: Weiye Xiao, name: Lei Yang
Published: 2024-08-01 17:59:27

Audio-driven 3D facial animation aims to map input audio to realistic facial
motion. Despite significant progress, limitations arise from inconsistent 3D
annotations, restricting previous models to training on specific annotations
and thereby constraining the training scale. In this work, we present
UniTalker, a unified model featuring a multi-head architecture designed to
effectively leverage datasets with varied annotations. To enhance training
stability and ensure consistency among multi-head outputs, we employ three
training strategies, namely, PCA, model warm-up, and pivot identity embedding.
To expand the training scale and diversity, we assemble A2F-Bench, comprising
five publicly available datasets and three newly curated datasets. These
datasets contain a wide range of audio domains, covering multilingual speech
voices and songs, thereby scaling the training data from commonly employed
datasets, typically less than 1 hour, to 18.5 hours. With a single trained
UniTalker model, we achieve substantial lip vertex error reductions of 9.2% for
BIWI dataset and 13.7% for Vocaset. Additionally, the pre-trained UniTalker
exhibits promise as the foundation model for audio-driven facial animation
tasks. Fine-tuning the pre-trained UniTalker on seen datasets further enhances
performance on each dataset, with an average error reduction of 6.3% on
A2F-Bench. Moreover, fine-tuning UniTalker on an unseen dataset with only half
the data surpasses prior state-of-the-art models trained on the full dataset.
The code and dataset are available at the project page
https://github.com/X-niper/UniTalker.
Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal
  Language Model
Authors: name: Benlin Liu, name: Yuhao Dong, name: Yiqin Wang, name: Yongming Rao, name: Yansong Tang, name: Wei-Chiu Ma, name: Ranjay Krishna
Published: 2024-08-01 17:57:12

Multimodal language models (MLLMs) are increasingly being implemented in
real-world environments, necessitating their ability to interpret 3D spaces and
comprehend temporal dynamics. Despite their potential, current top models
within our community still fall short in adequately understanding spatial and
temporal dimensions. We introduce Coarse Correspondence, a simple,
training-free, effective, and general-purpose visual prompting method to elicit
3D and temporal understanding in multimodal LLMs. Our method uses a lightweight
tracking model to find object correspondences between frames in a video or
between sets of image viewpoints. It selects the most frequent object instances
and visualizes them with markers with unique IDs in the image. With this simple
approach, we achieve state-of-the-art results on 3D understanding benchmarks
including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form
video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic
dataset to evaluate whether MLLMs can reason about space from a described
viewpoint other than the camera viewpoint. Again, Coarse Correspondence
improves spatial perspective-taking abilities but we highlight that MLLMs
struggle with this task. Together, we demonstrate that our simple prompting
method can significantly aid downstream tasks that require 3D or temporal
reasoning.
TurboEdit: Text-Based Image Editing Using Few-Step Diffusion Models
Authors: name: Gilad Deutch, name: Rinon Gal, name: Daniel Garibi, name: Or Patashnik, name: Daniel Cohen-Or
Published: 2024-08-01 17:27:28

Diffusion models have opened the path to a wide range of text-based image
editing frameworks. However, these typically build on the multi-step nature of
the diffusion backwards process, and adapting them to distilled, fast-sampling
methods has proven surprisingly challenging. Here, we focus on a popular line
of text-based editing frameworks - the ``edit-friendly'' DDPM-noise inversion
approach. We analyze its application to fast sampling methods and categorize
its failures into two classes: the appearance of visual artifacts, and
insufficient editing strength. We trace the artifacts to mismatched noise
statistics between inverted noises and the expected noise schedule, and suggest
a shifted noise schedule which corrects for this offset. To increase editing
strength, we propose a pseudo-guidance approach that efficiently increases the
magnitude of edits without introducing new artifacts. All in all, our method
enables text-based image editing with as few as three diffusion steps, while
providing novel insights into the mechanisms behind popular text-based editing
approaches.
SAM 2: Segment Anything in Images and Videos
Authors: name: Nikhila Ravi, name: Valentin Gabeur, name: Yuan-Ting Hu, name: Ronghang Hu, name: Chaitanya Ryali, name: Tengyu Ma, name: Haitham Khedr, name: Roman Rädle, name: Chloe Rolland, name: Laura Gustafson, name: Eric Mintun, name: Junting Pan, name: Kalyan Vasudev Alwala, name: Nicolas Carion, name: Chao-Yuan Wu, name: Ross Girshick, name: Piotr Dollár, name: Christoph Feichtenhofer
Published: 2024-08-01 17:00:08

We present Segment Anything Model 2 (SAM 2), a foundation model towards
solving promptable visual segmentation in images and videos. We build a data
engine, which improves model and data via user interaction, to collect the
largest video segmentation dataset to date. Our model is a simple transformer
architecture with streaming memory for real-time video processing. SAM 2
trained on our data provides strong performance across a wide range of tasks.
In video segmentation, we observe better accuracy, using 3x fewer interactions
than prior approaches. In image segmentation, our model is more accurate and 6x
faster than the Segment Anything Model (SAM). We believe that our data, model,
and insights will serve as a significant milestone for video segmentation and
related perception tasks. We are releasing a version of our model, the dataset
and an interactive demo.
Improving Text Embeddings for Smaller Language Models Using Contrastive
  Fine-tuning
Authors: name: Trapoom Ukarapol, name: Zhicheng Lee, name: Amy Xin
Published: 2024-08-01 16:31:35

While Large Language Models show remarkable performance in natural language
understanding, their resource-intensive nature makes them less accessible. In
contrast, smaller language models such as MiniCPM offer more sustainable
scalability, but often underperform without specialized optimization. In this
paper, we explore the enhancement of smaller language models through the
improvement of their text embeddings. We select three language models, MiniCPM,
Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our
results demonstrate that this fine-tuning method enhances the quality of text
embeddings for all three models across various benchmarks, with MiniCPM showing
the most significant improvements of an average 56.33\% performance gain. The
contrastive fine-tuning code is publicly available at
https://github.com/trapoom555/Language-Model-STS-CFT.
SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and
  Illumination Disentanglement
Authors: name: Mark Boss, name: Zixuan Huang, name: Aaryaman Vasishta, name: Varun Jampani
Published: 2024-08-01 15:41:57

We present SF3D, a novel method for rapid and high-quality textured object
mesh reconstruction from a single image in just 0.5 seconds. Unlike most
existing approaches, SF3D is explicitly trained for mesh generation,
incorporating a fast UV unwrapping technique that enables swift texture
generation rather than relying on vertex colors. The method also learns to
predict material parameters and normal maps to enhance the visual quality of
the reconstructed 3D meshes. Furthermore, SF3D integrates a delighting step to
effectively remove low-frequency illumination effects, ensuring that the
reconstructed meshes can be easily used in novel illumination conditions.
Experiments demonstrate the superior performance of SF3D over the existing
techniques. Project page: https://stable-fast-3d.github.io
Tails Tell Tales: Chapter-Wide Manga Transcriptions with Character Names
Authors: name: Ragav Sachdeva, name: Gyungin Shin, name: Andrew Zisserman
Published: 2024-08-01 05:47:04

Enabling engagement of manga by visually impaired individuals presents a
significant challenge due to its inherently visual nature. With the goal of
fostering accessibility, this paper aims to generate a dialogue transcript of a
complete manga chapter, entirely automatically, with a particular emphasis on
ensuring narrative consistency. This entails identifying (i) what is being
said, i.e., detecting the texts on each page and classifying them into
essential vs non-essential, and (ii) who is saying it, i.e., attributing each
dialogue to its speaker, while ensuring the same characters are named
consistently throughout the chapter.
  To this end, we introduce: (i) Magiv2, a model that is capable of generating
high-quality chapter-wide manga transcripts with named characters and
significantly higher precision in speaker diarisation over prior works; (ii) an
extension of the PopManga evaluation dataset, which now includes annotations
for speech-bubble tail boxes, associations of text to corresponding tails,
classifications of text as essential or non-essential, and the identity for
each character box; and (iii) a new character bank dataset, which comprises
over 11K characters from 76 manga series, featuring 11.5K exemplar character
images in total, as well as a list of chapters in which they appear. The code,
trained model, and both datasets can be found at:
https://github.com/ragavsachdeva/magi
Sentence-wise Speech Summarization: Task, Datasets, and End-to-End
  Modeling with LM Knowledge Distillation
Authors: name: Kohei Matsuura, name: Takanori Ashihara, name: Takafumi Moriya, name: Masato Mimura, name: Takatomo Kano, name: Atsunori Ogawa, name: Marc Delcroix
Published: 2024-08-01 00:18:21

This paper introduces a novel approach called sentence-wise speech
summarization (Sen-SSum), which generates text summaries from a spoken document
in a sentence-by-sentence manner. Sen-SSum combines the real-time processing of
automatic speech recognition (ASR) with the conciseness of speech
summarization. To explore this approach, we present two datasets for Sen-SSum:
Mega-SSum and CSJ-SSum. Using these datasets, our study evaluates two types of
Transformer-based models: 1) cascade models that combine ASR and strong text
summarization models, and 2) end-to-end (E2E) models that directly convert
speech into a text summary. While E2E models are appealing to develop
compute-efficient models, they perform worse than cascade models. Therefore, we
propose knowledge distillation for E2E models using pseudo-summaries generated
by the cascade models. Our experiments show that this proposed knowledge
distillation effectively improves the performance of the E2E model on both
datasets.
Improving 2D Feature Representations by 3D-Aware Fine-Tuning
Authors: name: Yuanwen Yue, name: Anurag Das, name: Francis Engelmann, name: Siyu Tang, name: Jan Eric Lenssen
Published: 2024-07-29 17:59:21

Current visual foundation models are trained purely on unstructured 2D data,
limiting their understanding of 3D structure of objects and scenes. In this
work, we show that fine-tuning on 3D-aware data improves the quality of
emerging semantic features. We design a method to lift semantic 2D features
into an efficient 3D Gaussian representation, which allows us to re-render them
for arbitrary views. Using the rendered 3D-aware features, we design a
fine-tuning strategy to transfer such 3D awareness into a 2D foundation model.
We demonstrate that models fine-tuned in that way produce features that readily
improve downstream task performance in semantic segmentation and depth
estimation through simple linear probing. Notably, though fined-tuned on a
single indoor dataset, the improvement is transferable to a variety of indoor
datasets and out-of-domain datasets. We hope our study encourages the community
to consider injecting 3D awareness when training 2D foundation models. Project
page: https://ywyue.github.io/FiT3D.
NeRF-MAE: Masked AutoEncoders for Self-Supervised 3D Representation
  Learning for Neural Radiance Fields
Authors: name: Muhammad Zubair Irshad, name: Sergey Zakharov, name: Vitor Guizilini, name: Adrien Gaidon, name: Zsolt Kira, name: Rares Ambrus
Published: 2024-04-01 17:59:55

Neural fields excel in computer vision and robotics due to their ability to
understand the 3D visual world such as inferring semantics, geometry, and
dynamics. Given the capabilities of neural fields in densely representing a 3D
scene from 2D images, we ask the question: Can we scale their self-supervised
pretraining, specifically using masked autoencoders, to generate effective 3D
representations from posed RGB images. Owing to the astounding success of
extending transformers to novel data modalities, we employ standard 3D Vision
Transformers to suit the unique formulation of NeRFs. We leverage NeRF's
volumetric grid as a dense input to the transformer, contrasting it with other
3D representations such as pointclouds where the information density can be
uneven, and the representation is irregular. Due to the difficulty of applying
masked autoencoders to an implicit representation, such as NeRF, we opt for
extracting an explicit representation that canonicalizes scenes across domains
by employing the camera trajectory for sampling. Our goal is made possible by
masking random patches from NeRF's radiance and density grid and employing a
standard 3D Swin Transformer to reconstruct the masked patches. In doing so,
the model can learn the semantic and spatial structure of complete scenes. We
pretrain this representation at scale on our proposed curated posed-RGB data,
totaling over 1.8 million images. Once pretrained, the encoder is used for
effective 3D transfer learning. Our novel self-supervised pretraining for
NeRFs, NeRF-MAE, scales remarkably well and improves performance on various
challenging 3D tasks. Utilizing unlabeled posed 2D data for pretraining,
NeRF-MAE significantly outperforms self-supervised 3D pretraining and NeRF
scene understanding baselines on Front3D and ScanNet datasets with an absolute
performance improvement of over 20% AP50 and 8% AP25 for 3D object detection.
TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization
  Methods
Authors: name: Gabriel Loiseau, name: Damien Sileo, name: Damien Riquet, name: Maxime Meyer, name: Marc Tommasi
Published: 2024-07-31 14:24:01

Authorship obfuscation aims to disguise the identity of an author within a
text by altering the writing style, vocabulary, syntax, and other linguistic
features associated with the text author. This alteration needs to balance
privacy and utility. While strong obfuscation techniques can effectively hide
the author's identity, they often degrade the quality and usefulness of the
text for its intended purpose. Conversely, maintaining high utility tends to
provide insufficient privacy, making it easier for an adversary to de-anonymize
the author. Thus, achieving an optimal trade-off between these two conflicting
objectives is crucial. In this paper, we propose TAROT: Task-Oriented
Authorship Obfuscation Using Policy Optimization, a new unsupervised authorship
obfuscation method whose goal is to optimize the privacy-utility trade-off by
regenerating the entire text considering its downstream utility. Our approach
leverages policy optimization as a fine-tuning paradigm over small language
models in order to rewrite texts by preserving author identity and downstream
task utility. We show that our approach largely reduce the accuracy of
attackers while preserving utility. We make our code and models publicly
available.
Open-Vocabulary Audio-Visual Semantic Segmentation
Authors: name: Ruohao Guo, name: Liao Qu, name: Dantong Niu, name: Yanyu Qi, name: Wenzhen Yue, name: Ji Shi, name: Bowei Xing, name: Xianghua Ying
Published: 2024-07-31 16:14:09

Audio-visual semantic segmentation (AVSS) aims to segment and classify
sounding objects in videos with acoustic cues. However, most approaches operate
on the close-set assumption and only identify pre-defined categories from
training data, lacking the generalization ability to detect novel categories in
practical applications. In this paper, we introduce a new task: open-vocabulary
audio-visual semantic segmentation, extending AVSS task to open-world scenarios
beyond the annotated label space. This is a more challenging task that requires
recognizing all categories, even those that have never been seen nor heard
during training. Moreover, we propose the first open-vocabulary AVSS framework,
OV-AVSS, which mainly consists of two parts: 1) a universal sound source
localization module to perform audio-visual fusion and locate all potential
sounding objects and 2) an open-vocabulary classification module to predict
categories with the help of the prior knowledge from large-scale pre-trained
vision-language models. To properly evaluate the open-vocabulary AVSS, we split
zero-shot training and testing subsets based on the AVSBench-semantic
benchmark, namely AVSBench-OV. Extensive experiments demonstrate the strong
segmentation and zero-shot generalization ability of our model on all
categories. On the AVSBench-OV dataset, OV-AVSS achieves 55.43% mIoU on base
categories and 29.14% mIoU on novel categories, exceeding the state-of-the-art
zero-shot method by 41.88%/20.61% and open-vocabulary method by 10.2%/11.6%.
The code is available at https://github.com/ruohaoguo/ovavss.
ShieldGemma: Generative AI Content Moderation Based on Gemma
Authors: name: Wenjun Zeng, name: Yuchi Liu, name: Ryan Mullins, name: Ludovic Peran, name: Joe Fernandez, name: Hamza Harkous, name: Karthik Narasimhan, name: Drew Proud, name: Piyush Kumar, name: Bhaktipriya Radharapu, name: Olivia Sturman, name: Oscar Wahltinez
Published: 2024-07-31 17:48:14

We present ShieldGemma, a comprehensive suite of LLM-based safety content
moderation models built upon Gemma2. These models provide robust,
state-of-the-art predictions of safety risks across key harm types (sexually
explicit, dangerous content, harassment, hate speech) in both user input and
LLM-generated output. By evaluating on both public and internal benchmarks, we
demonstrate superior performance compared to existing models, such as Llama
Guard (+10.8\% AU-PRC on public benchmarks) and WildCard (+4.3\%).
Additionally, we present a novel LLM-based data curation pipeline, adaptable to
a variety of safety-related tasks and beyond. We have shown strong
generalization performance for model trained mainly on synthetic data. By
releasing ShieldGemma, we provide a valuable resource to the research
community, advancing LLM safety and enabling the creation of more effective
content moderation solutions for developers.
Berkeley Humanoid: A Research Platform for Learning-based Control
Authors: name: Qiayuan Liao, name: Bike Zhang, name: Xuanyu Huang, name: Xiaoyu Huang, name: Zhongyu Li, name: Koushil Sreenath
Published: 2024-07-31 17:52:55

We introduce Berkeley Humanoid, a reliable and low-cost mid-scale humanoid
research platform for learning-based control. Our lightweight, in-house-built
robot is designed specifically for learning algorithms with low simulation
complexity, anthropomorphic motion, and high reliability against falls. The
robot's narrow sim-to-real gap enables agile and robust locomotion across
various terrains in outdoor environments, achieved with a simple reinforcement
learning controller using light domain randomization. Furthermore, we
demonstrate the robot traversing for hundreds of meters, walking on a steep
unpaved trail, and hopping with single and double legs as a testimony to its
high performance in dynamical walking. Capable of omnidirectional locomotion
and withstanding large perturbations with a compact setup, our system aims for
scalable, sim-to-real deployment of learning-based humanoid systems. Please
check http://berkeley-humanoid.com for more details.