Orionnnn/hub-stats huggingface dataset:
Making Text Embedders Few-Shot Learners
Authors: name: Chaofan Li, name: MingHao Qin, name: Shitao Xiao, name: Jianlyu Chen, name: Kun Luo, name: Yingxia Shao, name: Defu Lian, name: Zheng Liu
Published: 2024-09-24 03:30:19

Large language models (LLMs) with decoder-only architectures demonstrate
remarkable in-context learning (ICL) capabilities. This feature enables them to
effectively handle both familiar and novel tasks by utilizing examples provided
within their input context. Recognizing the potential of this capability, we
propose leveraging the ICL feature in LLMs to enhance the process of text
embedding generation. To this end, we introduce a novel model bge-en-icl, which
employs few-shot examples to produce high-quality text embeddings. Our approach
integrates task-related examples directly into the query side, resulting in
significant improvements across various tasks. Additionally, we have
investigated how to effectively utilize LLMs as embedding models, including
various attention mechanisms, pooling methods, etc. Our findings suggest that
retaining the original framework often yields the best results, underscoring
that simplicity is best. Experimental results on the MTEB and AIR-Bench
benchmarks demonstrate that our approach sets new state-of-the-art (SOTA)
performance. Our model, code and dataset are freely available at
https://github.com/FlagOpen/FlagEmbedding .
HelloBench: Evaluating Long Text Generation Capabilities of Large
  Language Models
Authors: name: Haoran Que, name: Feiyu Duan, name: Liqun He, name: Yutao Mou, name: Wangchunshu Zhou, name: Jiaheng Liu, name: Wenge Rong, name: Zekun Moore Wang, name: Jian Yang, name: Ge Zhang, name: Junran Peng, name: Zhaoxiang Zhang, name: Songyang Zhang, name: Kai Chen
Published: 2024-09-24 15:38:11

In recent years, Large Language Models (LLMs) have demonstrated remarkable
capabilities in various tasks (e.g., long-context understanding), and many
benchmarks have been proposed. However, we observe that long text generation
capabilities are not well investigated. Therefore, we introduce the
Hierarchical Long Text Generation Benchmark (HelloBench), a comprehensive,
in-the-wild, and open-ended benchmark to evaluate LLMs' performance in
generating long text. Based on Bloom's Taxonomy, HelloBench categorizes long
text generation tasks into five subtasks: open-ended QA, summarization, chat,
text completion, and heuristic text generation. Besides, we propose
Hierarchical Long Text Evaluation (HelloEval), a human-aligned evaluation
method that significantly reduces the time and effort required for human
evaluation while maintaining a high correlation with human evaluation. We have
conducted extensive experiments across around 30 mainstream LLMs and observed
that the current LLMs lack long text generation capabilities. Specifically,
first, regardless of whether the instructions include explicit or implicit
length constraints, we observe that most LLMs cannot generate text that is
longer than 4000 words. Second, we observe that while some LLMs can generate
longer text, many issues exist (e.g., severe repetition and quality
degradation). Third, to demonstrate the effectiveness of HelloEval, we compare
HelloEval with traditional metrics (e.g., ROUGE, BLEU, etc.) and LLM-as-a-Judge
methods, which show that HelloEval has the highest correlation with human
evaluation. We release our code in https://github.com/Quehry/HelloBench.
Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable
  Robot Manipulation
Authors: name: Homanga Bharadhwaj, name: Debidatta Dwibedi, name: Abhinav Gupta, name: Shubham Tulsiani, name: Carl Doersch, name: Ted Xiao, name: Dhruv Shah, name: Fei Xia, name: Dorsa Sadigh, name: Sean Kirmani
Published: 2024-09-24 17:57:33

How can robot manipulation policies generalize to novel tasks involving
unseen object types and new motions? In this paper, we provide a solution in
terms of predicting motion information from web data through human video
generation and conditioning a robot policy on the generated video. Instead of
attempting to scale robot data collection which is expensive, we show how we
can leverage video generation models trained on easily available web data, for
enabling generalization. Our approach Gen2Act casts language-conditioned
manipulation as zero-shot human video generation followed by execution with a
single policy conditioned on the generated video. To train the policy, we use
an order of magnitude less robot interaction data compared to what the video
prediction model was trained on. Gen2Act doesn't require fine-tuning the video
model at all and we directly use a pre-trained model for generating human
videos. Our results on diverse real-world scenarios show how Gen2Act enables
manipulating unseen object types and performing novel motions for tasks not
present in the robot data. Videos are at https://homangab.github.io/gen2act/
MonoFormer: One Transformer for Both Diffusion and Autoregression
Authors: name: Chuyang Zhao, name: Yuxing Song, name: Wenhao Wang, name: Haocheng Feng, name: Errui Ding, name: Yifan Sun, name: Xinyan Xiao, name: Jingdong Wang
Published: 2024-09-24 17:51:04

Most existing multimodality methods use separate backbones for
autoregression-based discrete text generation and diffusion-based continuous
visual generation, or the same backbone by discretizing the visual data to use
autoregression for both text and visual generation. In this paper, we propose
to study a simple idea: share one transformer for both autoregression and
diffusion. The feasibility comes from two main aspects: (i) Transformer is
successfully applied to diffusion for visual generation, and (ii) transformer
training for autoregression and diffusion is very similar, and the difference
merely lies in that diffusion uses bidirectional attention mask and
autoregression uses causal attention mask. Experimental results show that our
approach achieves comparable image generation performance to current
state-of-the-art methods as well as maintains the text generation capability.
The project is publicly available at https://monoformer.github.io/.
DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
Authors: name: Zichen Jeff Cui, name: Hengkai Pan, name: Aadhithya Iyer, name: Siddhant Haldar, name: Lerrel Pinto
Published: 2024-09-18 17:59:43

Imitation learning has proven to be a powerful tool for training complex
visuomotor policies. However, current methods often require hundreds to
thousands of expert demonstrations to handle high-dimensional visual
observations. A key reason for this poor data efficiency is that visual
representations are predominantly either pretrained on out-of-domain data or
trained directly through a behavior cloning objective. In this work, we present
DynaMo, a new in-domain, self-supervised method for learning visual
representations. Given a set of expert demonstrations, we jointly learn a
latent inverse dynamics model and a forward dynamics model over a sequence of
image embeddings, predicting the next frame in latent space, without
augmentations, contrastive sampling, or access to ground truth actions.
Importantly, DynaMo does not require any out-of-domain data such as Internet
datasets or cross-embodied datasets. On a suite of six simulated and real
environments, we show that representations learned with DynaMo significantly
improve downstream imitation learning performance over prior self-supervised
learning objectives, and pretrained representations. Gains from using DynaMo
hold across policy classes such as Behavior Transformer, Diffusion Policy, MLP,
and nearest neighbors. Finally, we ablate over key components of DynaMo and
measure its impact on downstream policy performance. Robot videos are best
viewed at https://dynamo-ssl.github.io
Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror
  Reflections
Authors: name: Ankit Dhiman, name: Manan Shah, name: Rishubh Parihar, name: Yash Bhalgat, name: Lokesh R Boregowda, name: R Venkatesh Babu
Published: 2024-09-23 02:59:07

We tackle the problem of generating highly realistic and plausible mirror
reflections using diffusion-based generative models. We formulate this problem
as an image inpainting task, allowing for more user control over the placement
of mirrors during the generation process. To enable this, we create SynMirror,
a large-scale dataset of diverse synthetic scenes with objects placed in front
of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D
objects, along with their associated depth maps, normal maps and instance-wise
segmentation masks, to capture relevant geometric properties of the scene.
Using this dataset, we propose a novel depth-conditioned inpainting method
called MirrorFusion, which generates high-quality geometrically consistent and
photo-realistic mirror reflections given an input image and a mask depicting
the mirror region. MirrorFusion outperforms state-of-the-art methods on
SynMirror, as demonstrated by extensive quantitative and qualitative analysis.
To the best of our knowledge, we are the first to successfully tackle the
challenging problem of generating controlled and faithful mirror reflections of
an object in a scene using diffusion based models. SynMirror and MirrorFusion
open up new avenues for image editing and augmented reality applications for
practitioners and researchers alike.
SpaceBlender: Creating Context-Rich Collaborative Spaces Through
  Generative 3D Scene Blending
Authors: name: Nels Numan, name: Shwetha Rajaram, name: Balasaravanan Thoravi Kumaravel, name: Nicolai Marquardt, name: Andrew D. Wilson
Published: 2024-09-20 22:27:31

There is increased interest in using generative AI to create 3D spaces for
Virtual Reality (VR) applications. However, today's models produce artificial
environments, falling short of supporting collaborative tasks that benefit from
incorporating the user's physical context. To generate environments that
support VR telepresence, we introduce SpaceBlender, a novel pipeline that
utilizes generative AI techniques to blend users' physical surroundings into
unified virtual spaces. This pipeline transforms user-provided 2D images into
context-rich 3D environments through an iterative process consisting of depth
estimation, mesh alignment, and diffusion-based space completion guided by
geometric priors and adaptive text prompts. In a preliminary within-subjects
study, where 20 participants performed a collaborative VR affinity diagramming
task in pairs, we compared SpaceBlender with a generic virtual environment and
a state-of-the-art scene generation framework, evaluating its ability to create
virtual spaces suitable for collaboration. Participants appreciated the
enhanced familiarity and context provided by SpaceBlender but also noted
complexities in the generative environments that could detract from task focus.
Drawing on participant feedback, we propose directions for improving the
pipeline and discuss the value and design of blended spaces for different
scenarios.
Style over Substance: Failure Modes of LLM Judges in Alignment
  Benchmarking
Authors: name: Benjamin Feuer, name: Micah Goldblum, name: Teresa Datta, name: Sanjana Nambiar, name: Raz Besaleli, name: Samuel Dooley, name: Max Cembalest, name: John P. Dickerson
Published: 2024-09-23 17:58:07

The release of ChatGPT in November 2022 sparked an explosion of interest in
post-training and an avalanche of new preference optimization (PO) methods.
These methods claim superior alignment by virtue of better correspondence with
human pairwise preferences, often measured by LLM judges. In this work, we
attempt to answer the following question -- do LLM-judge preferences translate
to progress on other, more concrete metrics for alignment, and if not, why not?
We define a concrete metric for alignment, and introduce SOS-Bench, the largest
standardized, reproducible LLM meta-benchmark to date. We find that (1)
LLM-judgments do not correlate with concrete measures of safety, world
knowledge, and instruction following; (2) LLM judges have powerful implicit
biases, prioritizing style over factuality and safety; and (3) the supervised
fine-tuning (SFT) stage of post-training, and not the PO stage, has the
greatest impact on alignment, with data scaling and prompt diversity as the
driving factors. Our codebase and complete results can be found at
https://github.com/penfever/sos-bench.
Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining
  for Clinical LLMs
Authors: name: Clément Christophe, name: Tathagata Raha, name: Svetlana Maslenkova, name: Muhammad Umar Salman, name: Praveen K Kanithi, name: Marco AF Pimentel, name: Shadab Khan
Published: 2024-09-23 13:09:57

Large Language Models (LLMs) have demonstrated significant potential in
transforming clinical applications. In this study, we investigate the efficacy
of four techniques in adapting LLMs for clinical use-cases: continuous
pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ
these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale
clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning
dataset of 500 million tokens. Our evaluation across various clinical tasks
reveals the impact of each technique. While continuous pretraining beyond 250
billion tokens yields marginal improvements on its own, it establishes a strong
foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to
enhance generation quality, surprisingly demonstrates additional gains on our
benchmark. Complex prompt engineering methods further enhance performance.
These findings show the importance of tailoring fine-tuning strategies and
exploring innovative techniques to optimize LLM performance in the clinical
domain.
PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language
  Instructions
Authors: name: Weifeng Lin, name: Xinyu Wei, name: Renrui Zhang, name: Le Zhuo, name: Shitian Zhao, name: Siyuan Huang, name: Junlin Xie, name: Yu Qiao, name: Peng Gao, name: Hongsheng Li
Published: 2024-09-23 17:59:46

This paper presents a versatile image-to-image visual assistant, PixWizard,
designed for image generation, manipulation, and translation based on free-from
language instructions. To this end, we tackle a variety of vision tasks into a
unified image-text-to-image generation framework and curate an Omni
Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction
templates in natural language, we comprehensively include a large set of
diverse vision tasks such as text-to-image generation, image restoration, image
grounding, dense image prediction, image editing, controllable generation,
inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers
(DiT) as our foundation model and extend its capabilities with a flexible any
resolution mechanism, enabling the model to dynamically process images based on
the aspect ratio of the input, closely aligning with human perceptual
processes. The model also incorporates structure-aware and semantic-aware
guidance to facilitate effective fusion of information from the input image.
Our experiments demonstrate that PixWizard not only shows impressive generative
and understanding abilities for images with diverse resolutions but also
exhibits promising generalization capabilities with unseen tasks and human
instructions. The code and related resources are available at
https://github.com/AFeng-x/PixWizard
Zero-shot Cross-lingual Voice Transfer for TTS
Authors: name: Fadi Biadsy, name: Youzheng Chen, name: Isaac Elias, name: Kyle Kastner, name: Gary Wang, name: Andrew Rosenberg, name: Bhuvana Ramabhadran
Published: 2024-09-20 21:27:13

In this paper, we introduce a zero-shot Voice Transfer (VT) module that can
be seamlessly integrated into a multi-lingual Text-to-speech (TTS) system to
transfer an individual's voice across languages. Our proposed VT module
comprises a speaker-encoder that processes reference speech, a bottleneck
layer, and residual adapters, connected to preexisting TTS layers. We compare
the performance of various configurations of these components and report Mean
Opinion Score (MOS) and Speaker Similarity across languages. Using a single
English reference speech per speaker, we achieve an average voice transfer
similarity score of 73% across nine target languages. Vocal characteristics
contribute significantly to the construction and perception of individual
identity. The loss of one's voice, due to physical or neurological conditions,
can lead to a profound sense of loss, impacting one's core identity. As a case
study, we demonstrate that our approach can not only transfer typical speech
but also restore the voices of individuals with dysarthria, even when only
atypical speech samples are available - a valuable utility for those who have
never had typical speech or banked their voice. Cross-lingual typical audio
samples, plus videos demonstrating voice restoration for dysarthric speakers
are available here
(google.github.io/tacotron/publications/zero_shot_voice_transfer).
MaterialFusion: Enhancing Inverse Rendering with Material Diffusion
  Priors
Authors: name: Yehonathan Litman, name: Or Patashnik, name: Kangle Deng, name: Aviral Agrawal, name: Rushikesh Zawar, name: Fernando De la Torre, name: Shubham Tulsiani
Published: 2024-09-23 17:59:06

Recent works in inverse rendering have shown promise in using multi-view
images of an object to recover shape, albedo, and materials. However, the
recovered components often fail to render accurately under new lighting
conditions due to the intrinsic challenge of disentangling albedo and material
properties from input images. To address this challenge, we introduce
MaterialFusion, an enhanced conventional 3D inverse rendering pipeline that
incorporates a 2D prior on texture and material properties. We present
StableMaterial, a 2D diffusion model prior that refines multi-lit data to
estimate the most likely albedo and material from given input appearances. This
model is trained on albedo, material, and relit image data derived from a
curated dataset of approximately ~12K artist-designed synthetic Blender objects
called BlenderVault. we incorporate this diffusion prior with an inverse
rendering framework where we use score distillation sampling (SDS) to guide the
optimization of the albedo and materials, improving relighting performance in
comparison with previous work. We validate MaterialFusion's relighting
performance on 4 datasets of synthetic and real objects under diverse
illumination conditions, showing our diffusion-aided approach significantly
improves the appearance of reconstructed objects under novel lighting
conditions. We intend to publicly release our BlenderVault dataset to support
further research in this field.
Phantom of Latent for Large Language and Vision Models
Authors: name: Byung-Kwan Lee, name: Sangyun Chung, name: Chae Won Kim, name: Beomchan Park, name: Yong Man Ro
Published: 2024-09-23 05:19:06

The success of visual instruction tuning has accelerated the development of
large language and vision models (LLVMs). Following the scaling laws of
instruction-tuned large language models (LLMs), LLVMs either have further
increased their sizes, reaching 26B, 34B, and even 80B parameters. While this
increase in model size has yielded significant performance gains, it demands
substantially more hardware resources for both training and inference.
Consequently, there naturally exists a strong need for efficient LLVMs that
achieve the performance of larger models while being smaller in size. To
achieve this need, we present a new efficient LLVM family with model sizes of
0.5B, 1.8B, 3.8B, and 7B parameters, Phantom, which significantly enhances
learning capabilities within limited structures. By temporarily increasing the
latent hidden dimension during multi-head self-attention (MHSA), we make LLVMs
prepare to look and understand much more vision-language knowledge on the
latent, without substantially increasing physical model sizes. To maximize its
advantage, we introduce Phantom Optimization (PO) using both autoregressive
supervised fine-tuning (SFT) and direct preference optimization (DPO)-like
concept, which effectively follows correct answers while eliminating incorrect
and ambiguous ones. Phantom outperforms numerous larger open- and closed-source
LLVMs, positioning itself as a leading solution in the landscape of efficient
LLVMs.
MaskedMimic: Unified Physics-Based Character Control Through Masked
  Motion Inpainting
Authors: name: Chen Tessler, name: Yunrong Guo, name: Ofir Nabati, name: Gal Chechik, name: Xue Bin Peng
Published: 2024-09-22 11:10:59

Crafting a single, versatile physics-based controller that can breathe life
into interactive characters across a wide spectrum of scenarios represents an
exciting frontier in character animation. An ideal controller should support
diverse control modalities, such as sparse target keyframes, text instructions,
and scene information. While previous works have proposed physically simulated,
scene-aware control models, these systems have predominantly focused on
developing controllers that each specializes in a narrow set of tasks and
control modalities. This work presents MaskedMimic, a novel approach that
formulates physics-based character control as a general motion inpainting
problem. Our key insight is to train a single unified model to synthesize
motions from partial (masked) motion descriptions, such as masked keyframes,
objects, text descriptions, or any combination thereof. This is achieved by
leveraging motion tracking data and designing a scalable training method that
can effectively utilize diverse motion descriptions to produce coherent
animations. Through this process, our approach learns a physics-based
controller that provides an intuitive control interface without requiring
tedious reward engineering for all behaviors of interest. The resulting
controller supports a wide range of control modalities and enables seamless
transitions between disparate tasks. By unifying character control through
motion inpainting, MaskedMimic creates versatile virtual characters. These
characters can dynamically adapt to complex scenes and compose diverse motions
on demand, enabling more interactive and immersive experiences.
RACER: Rich Language-Guided Failure Recovery Policies for Imitation
  Learning
Authors: name: Yinpei Dai, name: Jayjun Lee, name: Nima Fazeli, name: Joyce Chai
Published: 2024-09-23 02:50:33

Developing robust and correctable visuomotor policies for robotic
manipulation is challenging due to the lack of self-recovery mechanisms from
failures and the limitations of simple language instructions in guiding robot
actions. To address these issues, we propose a scalable data generation
pipeline that automatically augments expert demonstrations with failure
recovery trajectories and fine-grained language annotations for training. We
then introduce Rich languAge-guided failure reCovERy (RACER), a
supervisor-actor framework, which combines failure recovery data with rich
language descriptions to enhance robot control. RACER features a
vision-language model (VLM) that acts as an online supervisor, providing
detailed language guidance for error correction and task execution, and a
language-conditioned visuomotor policy as an actor to predict the next actions.
Our experimental results show that RACER outperforms the state-of-the-art
Robotic View Transformer (RVT) on RLbench across various evaluation settings,
including standard long-horizon tasks, dynamic goal-change tasks and zero-shot
unseen tasks, achieving superior performance in both simulated and real world
environments. Videos and code are available at:
https://rich-language-failure-recovery.github.io.