Orionnnn/hub-stats huggingface dataset:
Gecko: Versatile Text Embeddings Distilled from Large Language Models
Authors: name: Jinhyuk Lee, name: Zhuyun Dai, name: Xiaoqi Ren, name: Blair Chen, name: Daniel Cer, name: Jeremy R. Cole, name: Kai Hui, name: Michael Boratko, name: Rajvi Kapadia, name: Wen Ding, name: Yi Luan, name: Sai Meher Karthik Duddu, name: Gustavo Hernandez Abrego, name: Weiqiang Shi, name: Nithi Gupta, name: Aditya Kusupati, name: Prateek Jain, name: Siddhartha Reddy Jonnalagadda, name: Ming-Wei Chang, name: Iftekhar Naim
Published: 2024-03-29 17:56:40

We present Gecko, a compact and versatile text embedding model. Gecko
achieves strong retrieval performance by leveraging a key idea: distilling
knowledge from large language models (LLMs) into a retriever. Our two-step
distillation process begins with generating diverse, synthetic paired data
using an LLM. Next, we further refine the data quality by retrieving a set of
candidate passages for each query, and relabeling the positive and hard
negative passages using the same LLM. The effectiveness of our approach is
demonstrated by the compactness of the Gecko. On the Massive Text Embedding
Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing
entries with 768 embedding size. Gecko with 768 embedding dimensions achieves
an average score of 66.31, competing with 7x larger models and 5x higher
dimensional embeddings.
Jamba: A Hybrid Transformer-Mamba Language Model
Authors: name: Opher Lieber, name: Barak Lenz, name: Hofit Bata, name: Gal Cohen, name: Jhonathan Osin, name: Itay Dalmedigos, name: Erez Safahi, name: Shaked Meirom, name: Yonatan Belinkov, name: Shai Shalev-Shwartz, name: Omri Abend, name: Raz Alon, name: Tomer Asida, name: Amir Bergman, name: Roman Glozman, name: Michael Gokhman, name: Avashalom Manevich, name: Nir Ratner, name: Noam Rozen, name: Erez Shwartz, name: Mor Zusman, name: Yoav Shoham
Published: 2024-03-28 23:55:06

We present Jamba, a new base large language model based on a novel hybrid
Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba
interleaves blocks of Transformer and Mamba layers, enjoying the benefits of
both model families. MoE is added in some of these layers to increase model
capacity while keeping active parameter usage manageable. This flexible
architecture allows resource- and objective-specific configurations. In the
particular configuration we have implemented, we end up with a powerful model
that fits in a single 80GB GPU. Built at large scale, Jamba provides high
throughput and small memory footprint compared to vanilla Transformers, and at
the same time state-of-the-art performance on standard language model
benchmarks and long-context evaluations. Remarkably, the model presents strong
results for up to 256K tokens context length. We study various architectural
decisions, such as how to combine Transformer and Mamba layers, and how to mix
experts, and show that some of them are crucial in large scale modeling. We
also describe several interesting properties of these architectures which the
training and evaluation of Jamba have revealed, and plan to release checkpoints
from various ablation runs, to encourage further exploration of this novel
architecture. We make the weights of our implementation of Jamba publicly
available under a permissive license.
InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40
  Seconds
Authors: name: Zhiwen Fan, name: Wenyan Cong, name: Kairun Wen, name: Kevin Wang, name: Jian Zhang, name: Xinghao Ding, name: Danfei Xu, name: Boris Ivanovic, name: Marco Pavone, name: Georgios Pavlakos, name: Zhangyang Wang, name: Yue Wang
Published: 2024-03-29 17:29:58

While novel view synthesis (NVS) has made substantial progress in 3D computer
vision, it typically requires an initial estimation of camera intrinsics and
extrinsics from dense viewpoints. This pre-processing is usually conducted via
a Structure-from-Motion (SfM) pipeline, a procedure that can be slow and
unreliable, particularly in sparse-view scenarios with insufficient matched
features for accurate reconstruction. In this work, we integrate the strengths
of point-based representations (e.g., 3D Gaussian Splatting, 3D-GS) with
end-to-end dense stereo models (DUSt3R) to tackle the complex yet unresolved
issues in NVS under unconstrained settings, which encompasses pose-free and
sparse view challenges. Our framework, InstantSplat, unifies dense stereo
priors with 3D-GS to build 3D Gaussians of large-scale scenes from sparseview &
pose-free images in less than 1 minute. Specifically, InstantSplat comprises a
Coarse Geometric Initialization (CGI) module that swiftly establishes a
preliminary scene structure and camera parameters across all training views,
utilizing globally-aligned 3D point maps derived from a pre-trained dense
stereo pipeline. This is followed by the Fast 3D-Gaussian Optimization (F-3DGO)
module, which jointly optimizes the 3D Gaussian attributes and the initialized
poses with pose regularization. Experiments conducted on the large-scale
outdoor Tanks & Temples datasets demonstrate that InstantSplat significantly
improves SSIM (by 32%) while concurrently reducing Absolute Trajectory Error
(ATE) by 80%. These establish InstantSplat as a viable solution for scenarios
involving posefree and sparse-view conditions. Project page:
instantsplat.github.io.
Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for
  Reconstructing Challenging Surfaces
Authors: name: Mauro Comi, name: Alessio Tonioni, name: Max Yang, name: Jonathan Tremblay, name: Valts Blukis, name: Yijiong Lin, name: Nathan F. Lepora, name: Laurence Aitchison
Published: 2024-03-29 16:30:17

Touch and vision go hand in hand, mutually enhancing our ability to
understand the world. From a research perspective, the problem of mixing touch
and vision is underexplored and presents interesting challenges. To this end,
we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data
(local depth maps) with multi-view vision data to achieve surface
reconstruction and novel view synthesis. Our method optimises 3D Gaussian
primitives to accurately model the object's geometry at points of contact. By
creating a framework that decreases the transmittance at touch locations, we
achieve a refined surface reconstruction, ensuring a uniformly smooth depth
map. Touch is particularly useful when considering non-Lambertian objects (e.g.
shiny or reflective surfaces) since contemporary methods tend to fail to
reconstruct with fidelity specular highlights. By combining vision and tactile
sensing, we achieve more accurate geometry reconstructions with fewer images
than prior methods. We conduct evaluation on objects with glossy and reflective
surfaces and demonstrate the effectiveness of our approach, offering
significant improvements in reconstruction quality.
Unsolvable Problem Detection: Evaluating Trustworthiness of Vision
  Language Models
Authors: name: Atsuyuki Miyai, name: Jingkang Yang, name: Jingyang Zhang, name: Yifei Ming, name: Qing Yu, name: Go Irie, name: Yixuan Li, name: Hai Li, name: Ziwei Liu, name: Kiyoharu Aizawa
Published: 2024-03-29 17:59:53

This paper introduces a novel and significant challenge for Vision Language
Models (VLMs), termed Unsolvable Problem Detection (UPD). UPD examines the
VLM's ability to withhold answers when faced with unsolvable problems in the
context of Visual Question Answering (VQA) tasks. UPD encompasses three
distinct settings: Absent Answer Detection (AAD), Incompatible Answer Set
Detection (IASD), and Incompatible Visual Question Detection (IVQD). To deeply
investigate the UPD problem, extensive experiments indicate that most VLMs,
including GPT-4V and LLaVA-Next-34B, struggle with our benchmarks to varying
extents, highlighting significant room for the improvements. To address UPD, we
explore both training-free and training-based solutions, offering new insights
into their effectiveness and limitations. We hope our insights, together with
future efforts within the proposed UPD settings, will enhance the broader
understanding and development of more practical and reliable VLMs.
ReALM: Reference Resolution As Language Modeling
Authors: name: Joel Ruben Antony Moniz, name: Soundarya Krishnan, name: Melis Ozyildirim, name: Prathamesh Saraf, name: Halim Cagri Ates, name: Yuan Zhang, name: Hong Yu, name: Nidhi Rajshree
Published: 2024-03-29 17:59:06

Reference resolution is an important problem, one that is essential to
understand and successfully handle context of different kinds. This context
includes both previous turns and context that pertains to non-conversational
entities, such as entities on the user's screen or those running in the
background. While LLMs have been shown to be extremely powerful for a variety
of tasks, their use in reference resolution, particularly for
non-conversational entities, remains underutilized. This paper demonstrates how
LLMs can be used to create an extremely effective system to resolve references
of various types, by showing how reference resolution can be converted into a
language modeling problem, despite involving forms of entities like those on
screen that are not traditionally conducive to being reduced to a text-only
modality. We demonstrate large improvements over an existing system with
similar functionality across different types of references, with our smallest
model obtaining absolute gains of over 5% for on-screen references. We also
benchmark against GPT-3.5 and GPT-4, with our smallest model achieving
performance comparable to that of GPT-4, and our larger models substantially
outperforming it.
sDPO: Don't Use Your Data All at Once
Authors: name: Dahyun Kim, name: Yungi Kim, name: Wonho Song, name: Hyeonwoo Kim, name: Yunsu Kim, name: Sanghoon Kim, name: Chanjun Park
Published: 2024-03-28 09:56:04

As development of large language models (LLM) progresses, aligning them with
human preferences has become increasingly important. We propose stepwise DPO
(sDPO), an extension of the recently popularized direct preference optimization
(DPO) for alignment tuning. This approach involves dividing the available
preference datasets and utilizing them in a stepwise manner, rather than
employing it all at once. We demonstrate that this method facilitates the use
of more precisely aligned reference models within the DPO training framework.
Furthermore, sDPO trains the final model to be more performant, even
outperforming other popular LLMs with more parameters.
LITA: Language Instructed Temporal-Localization Assistant
Authors: name: De-An Huang, name: Shijia Liao, name: Subhashree Radhakrishnan, name: Hongxu Yin, name: Pavlo Molchanov, name: Zhiding Yu, name: Jan Kautz
Published: 2024-03-27 22:50:48

There has been tremendous progress in multimodal Large Language Models
(LLMs). Recent works have extended these models to video input with promising
instruction following capabilities. However, an important missing piece is
temporal localization. These models cannot accurately answer the "When?"
questions. We identify three key aspects that limit their temporal localization
capabilities: (i) time representation, (ii) architecture, and (iii) data. We
address these shortcomings by proposing Language Instructed
Temporal-Localization Assistant (LITA) with the following features: (1) We
introduce time tokens that encode timestamps relative to the video length to
better represent time in videos. (2) We introduce SlowFast tokens in the
architecture to capture temporal information at fine temporal resolution. (3)
We emphasize temporal localization data for LITA. In addition to leveraging
existing video datasets with timestamps, we propose a new task, Reasoning
Temporal Localization (RTL), along with the dataset, ActivityNet-RTL, for
learning and evaluating this task. Reasoning temporal localization requires
both the reasoning and temporal localization of Video LLMs. LITA demonstrates
strong performance on this challenging task, nearly doubling the temporal mean
intersection-over-union (mIoU) of baselines. In addition, we show that our
emphasis on temporal localization also substantially improves video-based text
generation compared to existing Video LLMs, including a 36% relative
improvement of Temporal Understanding. Code is available at:
https://github.com/NVlabs/LITA
GaussianCube: Structuring Gaussian Splatting using Optimal Transport for
  3D Generative Modeling
Authors: name: Bowen Zhang, name: Yiji Cheng, name: Jiaolong Yang, name: Chunyu Wang, name: Feng Zhao, name: Yansong Tang, name: Dong Chen, name: Baining Guo
Published: 2024-03-28 17:59:50

3D Gaussian Splatting (GS) have achieved considerable improvement over Neural
Radiance Fields in terms of 3D fitting fidelity and rendering speed. However,
this unstructured representation with scattered Gaussians poses a significant
challenge for generative modeling. To address the problem, we introduce
GaussianCube, a structured GS representation that is both powerful and
efficient for generative modeling. We achieve this by first proposing a
modified densification-constrained GS fitting algorithm which can yield
high-quality fitting results using a fixed number of free Gaussians, and then
re-arranging the Gaussians into a predefined voxel grid via Optimal Transport.
The structured grid representation allows us to use standard 3D U-Net as our
backbone in diffusion generative modeling without elaborate designs. Extensive
experiments conducted on ShapeNet and OmniObject3D show that our model achieves
state-of-the-art generation results both qualitatively and quantitatively,
underscoring the potential of GaussianCube as a powerful and versatile 3D
representation.
Mesh2NeRF: Direct Mesh Supervision for Neural Radiance Field
  Representation and Generation
Authors: name: Yujin Chen, name: Yinyu Nie, name: Benjamin Ummenhofer, name: Reiner Birkl, name: Michael Paulitsch, name: Matthias Müller, name: Matthias Nießner
Published: 2024-03-28 11:22:53

We present Mesh2NeRF, an approach to derive ground-truth radiance fields from
textured meshes for 3D generation tasks. Many 3D generative approaches
represent 3D scenes as radiance fields for training. Their ground-truth
radiance fields are usually fitted from multi-view renderings from a
large-scale synthetic 3D dataset, which often results in artifacts due to
occlusions or under-fitting issues. In Mesh2NeRF, we propose an analytic
solution to directly obtain ground-truth radiance fields from 3D meshes,
characterizing the density field with an occupancy function featuring a defined
surface thickness, and determining view-dependent color through a reflection
function considering both the mesh and environment lighting. Mesh2NeRF extracts
accurate radiance fields which provides direct supervision for training
generative NeRFs and single scene representation. We validate the effectiveness
of Mesh2NeRF across various tasks, achieving a noteworthy 3.12dB improvement in
PSNR for view synthesis in single scene representation on the ABO dataset, a
0.69 PSNR enhancement in the single-view conditional generation of ShapeNet
Cars, and notably improved mesh extraction from NeRF in the unconditional
generation of Objaverse Mugs.
TextCraftor: Your Text Encoder Can be Image Quality Controller
Authors: name: Yanyu Li, name: Xian Liu, name: Anil Kag, name: Ju Hu, name: Yerlan Idelbayev, name: Dhritiman Sagar, name: Yanzhi Wang, name: Sergey Tulyakov, name: Jian Ren
Published: 2024-03-27 19:52:55

Diffusion-based text-to-image generative models, e.g., Stable Diffusion, have
revolutionized the field of content generation, enabling significant
advancements in areas like image editing and video synthesis. Despite their
formidable capabilities, these models are not without their limitations. It is
still challenging to synthesize an image that aligns well with the input text,
and multiple runs with carefully crafted prompts are required to achieve
satisfactory results. To mitigate these limitations, numerous studies have
endeavored to fine-tune the pre-trained diffusion models, i.e., UNet, utilizing
various technologies. Yet, amidst these efforts, a pivotal question of
text-to-image diffusion model training has remained largely unexplored: Is it
possible and feasible to fine-tune the text encoder to improve the performance
of text-to-image diffusion models? Our findings reveal that, instead of
replacing the CLIP text encoder used in Stable Diffusion with other large
language models, we can enhance it through our proposed fine-tuning approach,
TextCraftor, leading to substantial improvements in quantitative benchmarks and
human assessments. Interestingly, our technique also empowers controllable
image generation through the interpolation of different text encoders
fine-tuned with various rewards. We also demonstrate that TextCraftor is
orthogonal to UNet finetuning, and can be combined to further improve
generative quality.
EgoLifter: Open-world 3D Segmentation for Egocentric Perception
Authors: name: Qiao Gu, name: Zhaoyang Lv, name: Duncan Frost, name: Simon Green, name: Julian Straub, name: Chris Sweeney
Published: 2024-03-26 21:48:27

In this paper we present EgoLifter, a novel system that can automatically
segment scenes captured from egocentric sensors into a complete decomposition
of individual 3D objects. The system is specifically designed for egocentric
data where scenes contain hundreds of objects captured from natural
(non-scanning) motion. EgoLifter adopts 3D Gaussians as the underlying
representation of 3D scenes and objects and uses segmentation masks from the
Segment Anything Model (SAM) as weak supervision to learn flexible and
promptable definitions of object instances free of any specific object
taxonomy. To handle the challenge of dynamic objects in ego-centric videos, we
design a transient prediction module that learns to filter out dynamic objects
in the 3D reconstruction. The result is a fully automatic pipeline that is able
to reconstruct 3D object instances as collections of 3D Gaussians that
collectively compose the entire scene. We created a new benchmark on the Aria
Digital Twin dataset that quantitatively demonstrates its state-of-the-art
performance in open-world 3D segmentation from natural egocentric input. We run
EgoLifter on various egocentric activity datasets which shows the promise of
the method for 3D egocentric perception at scale.
ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object
  Removal and Insertion
Authors: name: Daniel Winter, name: Matan Cohen, name: Shlomi Fruchter, name: Yael Pritch, name: Alex Rav-Acha, name: Yedid Hoshen
Published: 2024-03-27 17:59:52

Diffusion models have revolutionized image editing but often generate images
that violate physical laws, particularly the effects of objects on the scene,
e.g., occlusions, shadows, and reflections. By analyzing the limitations of
self-supervised approaches, we propose a practical solution centered on a
counterfactual dataset. Our method involves capturing a scene before and
after removing a single object, while minimizing other changes. By fine-tuning
a diffusion model on this dataset, we are able to not only remove objects but
also their effects on the scene. However, we find that applying this approach
for photorealistic object insertion requires an impractically large dataset. To
tackle this challenge, we propose bootstrap supervision; leveraging our object
removal model trained on a small counterfactual dataset, we synthetically
expand this dataset considerably. Our approach significantly outperforms prior
methods in photorealistic object removal and insertion, particularly at
modeling the effects of objects on the scene.
Garment3DGen: 3D Garment Stylization and Texture Generation
Authors: name: Nikolaos Sarafianos, name: Tuur Stuyck, name: Xiaoyu Xiang, name: Yilei Li, name: Jovan Popovic, name: Rakesh Ranjan
Published: 2024-03-27 17:59:33

We introduce Garment3DGen a new method to synthesize 3D garment assets from a
base mesh given a single input image as guidance. Our proposed approach allows
users to generate 3D textured clothes based on both real and synthetic images,
such as those generated by text prompts. The generated assets can be directly
draped and simulated on human bodies. First, we leverage the recent progress of
image to 3D diffusion methods to generate 3D garment geometries. However, since
these geometries cannot be utilized directly for downstream tasks, we propose
to use them as pseudo ground-truth and set up a mesh deformation optimization
procedure that deforms a base template mesh to match the generated 3D target.
Second, we introduce carefully designed losses that allow the input base mesh
to freely deform towards the desired target, yet preserve mesh quality and
topology such that they can be simulated. Finally, a texture estimation module
generates high-fidelity texture maps that are globally and locally consistent
and faithfully capture the input guidance, allowing us to render the generated
3D assets. With Garment3DGen users can generate the textured 3D garment of
their choice without the need of artist intervention. One can provide a textual
prompt describing the garment they desire to generate a simulation-ready 3D
asset. We present a plethora of quantitative and qualitative comparisons on
various assets both real and generated and provide use-cases of how one can
generate simulation-ready 3D garments.
FlexEdit: Flexible and Controllable Diffusion-based Object-centric Image
  Editing
Authors: name: Trong-Tung Nguyen, name: Duc-Anh Nguyen, name: Anh Tran, name: Cuong Pham
Published: 2024-03-27 14:24:30

Our work addresses limitations seen in previous approaches for object-centric
editing problems, such as unrealistic results due to shape discrepancies and
limited control in object replacement or insertion. To this end, we introduce
FlexEdit, a flexible and controllable editing framework for objects where we
iteratively adjust latents at each denoising step using our FlexEdit block.
Initially, we optimize latents at test time to align with specified object
constraints. Then, our framework employs an adaptive mask, automatically
extracted during denoising, to protect the background while seamlessly blending
new content into the target image. We demonstrate the versatility of FlexEdit
in various object editing tasks and curate an evaluation test suite with
samples from both real and synthetic images, along with novel evaluation
metrics designed for object-centric editing. We conduct extensive experiments
on different editing scenarios, demonstrating the superiority of our editing
framework over recent advanced text-guided image editing methods. Our project
page is published at https://flex-edit.github.io/.
Towards a World-English Language Model for On-Device Virtual Assistants
Authors: name: Rricha Jalota, name: Lyan Verwimp, name: Markus Nussbaum-Thom, name: Amr Mousa, name: Arturo Argueta, name: Youssef Oualil
Published: 2024-03-27 17:31:39

Neural Network Language Models (NNLMs) for Virtual Assistants (VAs) are
generally language-, region-, and in some cases, device-dependent, which
increases the effort to scale and maintain them. Combining NNLMs for one or
more of the categories is one way to improve scalability. In this work, we
combine regional variants of English to build a ``World English'' NNLM for
on-device VAs. In particular, we investigate the application of adapter
bottlenecks to model dialect-specific characteristics in our existing
production NNLMs {and enhance the multi-dialect baselines}. We find that
adapter modules are more effective in modeling dialects than specializing
entire sub-networks. Based on this insight and leveraging the design of our
production models, we introduce a new architecture for World English NNLM that
meets the accuracy, latency, and memory constraints of our single-dialect
models.