Orionnnn/hub-stats huggingface dataset:
INDUS: Effective and Efficient Language Models for Scientific
  Applications
Authors: name: Bishwaranjan Bhattacharjee, name: Aashka Trivedi, name: Masayasu Muraoka, name: Muthukumaran Ramasubramanian, name: Takuma Udagawa, name: Iksha Gurung, name: Rong Zhang, name: Bharath Dandala, name: Rahul Ramachandran, name: Manil Maskey, name: Kayleen Bugbee, name: Mike Little, name: Elizabeth Fancher, name: Lauren Sanders, name: Sylvain Costes, name: Sergi Blanco-Cuaresma, name: Kelly Lockhart, name: Thomas Allen, name: Felix Grazes, name: Megan Ansdel, name: Alberto Accomazzi, name: Yousef El-Kurdi, name: Davis Wertheimer, name: Birgit Pfitzmann, name: Cesar Berrospi Ramis, name: Michele Dolfi, name: Rafael Teixeira de Lima, name: Panos Vegenas, name: S. Karthik Mukkavilli, name: Peter Staar, name: Sanaz Vahidinia, name: Ryan McGranaghan, name: Armin Mehrabian, name: Tsendgar Lee
Published: 2024-05-17 12:15:07

Large language models (LLMs) trained on general domain corpora showed
remarkable results on natural language processing (NLP) tasks. However,
previous research demonstrated LLMs trained using domain-focused corpora
perform better on specialized tasks. Inspired by this pivotal insight, we
developed INDUS, a comprehensive suite of LLMs tailored for the Earth science,
biology, physics, heliophysics, planetary sciences and astrophysics domains and
trained using curated scientific corpora drawn from diverse data sources. The
suite of models include: (1) an encoder model trained using domain-specific
vocabulary and corpora to address natural language understanding tasks, (2) a
contrastive-learning-based general text embedding model trained using a diverse
set of datasets drawn from multiple sources to address information retrieval
tasks and (3) smaller versions of these models created using knowledge
distillation techniques to address applications which have latency or resource
constraints. We also created three new scientific benchmark datasets namely,
CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR
(IR) to accelerate research in these multi-disciplinary fields. Finally, we
show that our models outperform both general-purpose encoders (RoBERTa) and
existing domain-specific encoders (SciBERT) on these new tasks as well as
existing benchmark tasks in the domains of interest.
Dynamic data sampler for cross-language transfer learning in large
  language models
Authors: name: Yudong Li, name: Yuhao Feng, name: Wen Zhou, name: Zhe Zhao, name: Linlin Shen, name: Cheng Hou, name: Xianxu Hou
Published: 2024-05-17 08:40:51

Large Language Models (LLMs) have gained significant attention in the field
of natural language processing (NLP) due to their wide range of applications.
However, training LLMs for languages other than English poses significant
challenges, due to the difficulty in acquiring large-scale corpus and the
requisite computing resources. In this paper, we propose ChatFlow, a
cross-language transfer-based LLM, to address these challenges and train large
Chinese language models in a cost-effective manner. We employ a mix of Chinese,
English, and parallel corpus to continuously train the LLaMA2 model, aiming to
align cross-language representations and facilitate the knowledge transfer
specifically to the Chinese language model. In addition, we use a dynamic data
sampler to progressively transition the model from unsupervised pre-training to
supervised fine-tuning. Experimental results demonstrate that our approach
accelerates model convergence and achieves superior performance. We evaluate
ChatFlow on popular Chinese and English benchmarks, the results indicate that
it outperforms other Chinese models post-trained on LLaMA-2-7B.
Grounded 3D-LLM with Referent Tokens
Authors: name: Yilun Chen, name: Shuai Yang, name: Haifeng Huang, name: Tai Wang, name: Ruiyuan Lyu, name: Runsen Xu, name: Dahua Lin, name: Jiangmiao Pang
Published: 2024-05-16 18:03:41

Prior studies on 3D scene understanding have primarily developed specialized
models for specific tasks or required task-specific fine-tuning. In this study,
we propose Grounded 3D-LLM, which explores the potential of 3D large
multi-modal models (3D LMMs) to consolidate various 3D vision tasks within a
unified generative framework. The model uses scene referent tokens as special
noun phrases to reference 3D scenes, enabling the handling of sequences that
interleave 3D and textual data. It offers a natural approach for translating 3D
vision tasks into language formats using task-specific instruction templates.
To facilitate the use of referent tokens in subsequent language modeling, we
have curated large-scale grounded language datasets that offer finer scene-text
correspondence at the phrase level by bootstrapping existing object labels.
Subsequently, we introduced Contrastive LAnguage-Scene Pre-training (CLASP) to
effectively leverage this data, thereby integrating 3D vision with language
models. Our comprehensive evaluation covers open-ended tasks like dense
captioning and 3D QA, alongside close-ended tasks such as object detection and
language grounding. Experiments across multiple 3D benchmarks reveal the
leading performance and the broad applicability of Grounded 3D-LLM. Code and
datasets will be released on the project page:
https://groundedscenellm.github.io/grounded_3d-llm.github.io.
Layer-Condensed KV Cache for Efficient Inference of Large Language
  Models
Authors: name: Haoyi Wu, name: Kewei Tu
Published: 2024-05-17 08:59:46

Huge memory consumption has been a major bottleneck for deploying
high-throughput large language models in real-world applications. In addition
to the large number of parameters, the key-value (KV) cache for the attention
mechanism in the transformer architecture consumes a significant amount of
memory, especially when the number of layers is large for deep language models.
In this paper, we propose a novel method that only computes and caches the KVs
of a small number of layers, thus significantly saving memory consumption and
improving inference throughput. Our experiments on large language models show
that our method achieves up to 26times higher throughput than standard
transformers and competitive performance in language modeling and downstream
tasks. In addition, our method is orthogonal to existing transformer
memory-saving techniques, so it is straightforward to integrate them with our
model, achieving further improvement in inference efficiency. Our code is
available at https://github.com/whyNLP/LCKV.
Observational Scaling Laws and the Predictability of Language Model
  Performance
Authors: name: Yangjun Ruan, name: Chris J. Maddison, name: Tatsunori Hashimoto
Published: 2024-05-17 17:49:44

Understanding how language model performance varies with scale is critical to
benchmark and algorithm development. Scaling laws are one approach to building
this understanding, but the requirement of training models across many
different scales has limited their use. We propose an alternative,
observational approach that bypasses model training and instead builds scaling
laws from ~80 publically available models. Building a single scaling law from
multiple model families is challenging due to large variations in their
training compute efficiencies and capabilities. However, we show that these
variations are consistent with a simple, generalized scaling law where language
model performance is a function of a low-dimensional capability space, and
model families only vary in their efficiency in converting training compute to
capabilities. Using this approach, we show the surprising predictability of
complex scaling phenomena: we show that several emergent phenomena follow a
smooth, sigmoidal behavior and are predictable from small models; we show that
the agent performance of models such as GPT-4 can be precisely predicted from
simpler non-agentic benchmarks; and we show how to predict the impact of
post-training interventions like Chain-of-Thought and Self-Consistency as
language model capabilities continue to improve.
Toon3D: Seeing Cartoons from a New Perspective
Authors: name: Ethan Weber, name: Riley Peterlinz, name: Rohan Mathur, name: Frederik Warburg, name: Alexei A. Efros, name: Angjoo Kanazawa
Published: 2024-05-16 17:59:51

In this work, we recover the underlying 3D structure of non-geometrically
consistent scenes. We focus our analysis on hand-drawn images from cartoons and
anime. Many cartoons are created by artists without a 3D rendering engine,
which means that any new image of a scene is hand-drawn. The hand-drawn images
are usually faithful representations of the world, but only in a qualitative
sense, since it is difficult for humans to draw multiple perspectives of an
object or scene 3D consistently. Nevertheless, people can easily perceive 3D
scenes from inconsistent inputs! In this work, we correct for 2D drawing
inconsistencies to recover a plausible 3D structure such that the newly warped
drawings are consistent with each other. Our pipeline consists of a
user-friendly annotation tool, camera pose estimation, and image deformation to
recover a dense structure. Our method warps images to obey a perspective camera
model, enabling our aligned results to be plugged into novel-view synthesis
reconstruction methods to experience cartoons from viewpoints never drawn
before. Our project page is https://toon3d.studio/.
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction
Authors: name: Yunfan Jiang, name: Chen Wang, name: Ruohan Zhang, name: Jiajun Wu, name: Li Fei-Fei
Published: 2024-05-16 17:59:07

Learning in simulation and transferring the learned policy to the real world
has the potential to enable generalist robots. The key challenge of this
approach is to address simulation-to-reality (sim-to-real) gaps. Previous
methods often require domain-specific knowledge a priori. We argue that a
straightforward way to obtain such knowledge is by asking humans to observe and
assist robot policy execution in the real world. The robots can then learn from
humans to close various sim-to-real gaps. We propose TRANSIC, a data-driven
approach to enable successful sim-to-real transfer based on a human-in-the-loop
framework. TRANSIC allows humans to augment simulation policies to overcome
various unmodeled sim-to-real gaps holistically through intervention and online
correction. Residual policies can be learned from human corrections and
integrated with simulation policies for autonomous execution. We show that our
approach can achieve successful sim-to-real transfer in complex and
contact-rich manipulation tasks such as furniture assembly. Through synergistic
integration of policies learned in simulation and from humans, TRANSIC is
effective as a holistic approach to addressing various, often coexisting
sim-to-real gaps. It displays attractive properties such as scaling with human
effort. Videos and code are available at https://transic-robot.github.io/
Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection
Authors: name: Tianhe Ren, name: Qing Jiang, name: Shilong Liu, name: Zhaoyang Zeng, name: Wenlong Liu, name: Han Gao, name: Hongjie Huang, name: Zhengyu Ma, name: Xiaoke Jiang, name: Yihao Chen, name: Yuda Xiong, name: Hao Zhang, name: Feng Li, name: Peijun Tang, name: Kent Yu, name: Lei Zhang
Published: 2024-05-16 17:54:15

This paper introduces Grounding DINO 1.5, a suite of advanced open-set object
detection models developed by IDEA Research, which aims to advance the "Edge"
of open-set object detection. The suite encompasses two models: Grounding DINO
1.5 Pro, a high-performance model designed for stronger generalization
capability across a wide range of scenarios, and Grounding DINO 1.5 Edge, an
efficient model optimized for faster speed demanded in many applications
requiring edge deployment. The Grounding DINO 1.5 Pro model advances its
predecessor by scaling up the model architecture, integrating an enhanced
vision backbone, and expanding the training dataset to over 20 million images
with grounding annotations, thereby achieving a richer semantic understanding.
The Grounding DINO 1.5 Edge model, while designed for efficiency with reduced
feature scales, maintains robust detection capabilities by being trained on the
same comprehensive dataset. Empirical results demonstrate the effectiveness of
Grounding DINO 1.5, with the Grounding DINO 1.5 Pro model attaining a 54.3 AP
on the COCO detection benchmark and a 55.7 AP on the LVIS-minival zero-shot
transfer benchmark, setting new records for open-set object detection.
Furthermore, the Grounding DINO 1.5 Edge model, when optimized with TensorRT,
achieves a speed of 75.2 FPS while attaining a zero-shot performance of 36.2 AP
on the LVIS-minival benchmark, making it more suitable for edge computing
scenarios. Model examples and demos with API will be released at
https://github.com/IDEA-Research/Grounding-DINO-1.5-API
CAT3D: Create Anything in 3D with Multi-View Diffusion Models
Authors: name: Ruiqi Gao, name: Aleksander Holynski, name: Philipp Henzler, name: Arthur Brussee, name: Ricardo Martin-Brualla, name: Pratul Srinivasan, name: Jonathan T. Barron, name: Ben Poole
Published: 2024-05-16 17:59:05

Advances in 3D reconstruction have enabled high-quality 3D capture, but
require a user to collect hundreds to thousands of images to create a 3D scene.
We present CAT3D, a method for creating anything in 3D by simulating this
real-world capture process with a multi-view diffusion model. Given any number
of input images and a set of target novel viewpoints, our model generates
highly consistent novel views of a scene. These generated views can be used as
input to robust 3D reconstruction techniques to produce 3D representations that
can be rendered from any viewpoint in real-time. CAT3D can create entire 3D
scenes in as little as one minute, and outperforms existing methods for single
image and few-view 3D scene creation. See our project page for results and
interactive demos at https://cat3d.github.io .
LoRA Learns Less and Forgets Less
Authors: name: Dan Biderman, name: Jose Gonzalez Ortiz, name: Jacob Portes, name: Mansheej Paul, name: Philip Greengard, name: Connor Jennings, name: Daniel King, name: Sam Havens, name: Vitaliy Chiley, name: Jonathan Frankle, name: Cody Blakeney, name: John P. Cunningham
Published: 2024-05-15 19:27:45

Low-Rank Adaptation (LoRA) is a widely-used parameter-efficient finetuning
method for large language models. LoRA saves memory by training only low rank
perturbations to selected weight matrices. In this work, we compare the
performance of LoRA and full finetuning on two target domains, programming and
mathematics. We consider both the instruction finetuning (approx100K
prompt-response pairs) and continued pretraining (approx10B unstructured
tokens) data regimes. Our results show that, in most settings, LoRA
substantially underperforms full finetuning. Nevertheless, LoRA exhibits a
desirable form of regularization: it better maintains the base model's
performance on tasks outside the target domain. We show that LoRA provides
stronger regularization compared to common techniques such as weight decay and
dropout; it also helps maintain more diverse generations. We show that full
finetuning learns perturbations with a rank that is 10-100X greater than
typical LoRA configurations, possibly explaining some of the reported gaps. We
conclude by proposing best practices for finetuning with LoRA.
Many-Shot In-Context Learning in Multimodal Foundation Models
Authors: name: Yixing Jiang, name: Jeremy Irvin, name: Ji Hun Wang, name: Muhammad Ahmed Chaudhry, name: Jonathan H. Chen, name: Andrew Y. Ng
Published: 2024-05-16 04:02:43

Large language models are well-known to be effective at few-shot in-context
learning (ICL). Recent advancements in multimodal foundation models have
enabled unprecedentedly long context windows, presenting an opportunity to
explore their capability to perform ICL with many more demonstrating examples.
In this work, we evaluate the performance of multimodal foundation models
scaling from few-shot to many-shot ICL. We benchmark GPT-4o and Gemini 1.5 Pro
across 10 datasets spanning multiple domains (natural imagery, medical imagery,
remote sensing, and molecular imagery) and tasks (multi-class, multi-label, and
fine-grained classification). We observe that many-shot ICL, including up to
almost 2,000 multimodal demonstrating examples, leads to substantial
improvements compared to few-shot (<100 examples) ICL across all of the
datasets. Further, Gemini 1.5 Pro performance continues to improve log-linearly
up to the maximum number of tested examples on many datasets. Given the high
inference costs associated with the long prompts required for many-shot ICL, we
also explore the impact of batching multiple queries in a single API call. We
show that batching up to 50 queries can lead to performance improvements under
zero-shot and many-shot ICL, with substantial gains in the zero-shot setting on
multiple datasets, while drastically reducing per-query cost and latency.
Finally, we measure ICL data efficiency of the models, or the rate at which the
models learn from more demonstrating examples. We find that while GPT-4o and
Gemini 1.5 Pro achieve similar zero-shot performance across the datasets,
Gemini 1.5 Pro exhibits higher ICL data efficiency than GPT-4o on most
datasets. Our results suggest that many-shot ICL could enable users to
efficiently adapt multimodal foundation models to new applications and domains.
Our codebase is publicly available at
https://github.com/stanfordmlgroup/ManyICL .
Dual3D: Efficient and Consistent Text-to-3D Generation with Dual-mode
  Multi-view Latent Diffusion
Authors: name: Xinyang Li, name: Zhangyu Lai, name: Linning Xu, name: Jianfei Guo, name: Liujuan Cao, name: Shengchuan Zhang, name: Bo Dai, name: Rongrong Ji
Published: 2024-05-16 07:50:02

We present Dual3D, a novel text-to-3D generation framework that generates
high-quality 3D assets from texts in only 1 minute.The key component is a
dual-mode multi-view latent diffusion model. Given the noisy multi-view
latents, the 2D mode can efficiently denoise them with a single latent
denoising network, while the 3D mode can generate a tri-plane neural surface
for consistent rendering-based denoising. Most modules for both modes are tuned
from a pre-trained text-to-image latent diffusion model to circumvent the
expensive cost of training from scratch. To overcome the high rendering cost
during inference, we propose the dual-mode toggling inference strategy to use
only 1/10 denoising steps with 3D mode, successfully generating a 3D asset in
just 10 seconds without sacrificing quality. The texture of the 3D asset can
be further enhanced by our efficient texture refinement process in a short
time. Extensive experiments demonstrate that our method delivers
state-of-the-art performance while significantly reducing generation time. Our
project page is available at https://dual3d.github.io
Chameleon: Mixed-Modal Early-Fusion Foundation Models
Authors: name: Chameleon Team
Published: 2024-05-16 05:23:41

We present Chameleon, a family of early-fusion token-based mixed-modal models
capable of understanding and generating images and text in any arbitrary
sequence. We outline a stable training approach from inception, an alignment
recipe, and an architectural parameterization tailored for the early-fusion,
token-based, mixed-modal setting. The models are evaluated on a comprehensive
range of tasks, including visual question answering, image captioning, text
generation, image generation, and long-form mixed modal generation. Chameleon
demonstrates broad and general capabilities, including state-of-the-art
performance in image captioning tasks, outperforms Llama-2 in text-only tasks
while being competitive with models such as Mixtral 8x7B and Gemini-Pro, and
performs non-trivial image generation, all in a single model. It also matches
or exceeds the performance of much larger models, including Gemini Pro and
GPT-4V, according to human judgments on a new long-form mixed-modal generation
evaluation, where either the prompt or outputs contain mixed sequences of both
images and text. Chameleon marks a significant step forward in a unified
modeling of full multimodal documents.
Naturalistic Music Decoding from EEG Data via Latent Diffusion Models
Authors: name: Emilian Postolache, name: Natalia Polouliakh, name: Hiroaki Kitano, name: Akima Connelly, name: Emanuele Rodolà, name: Taketo Akama
Published: 2024-05-15 03:26:01

In this article, we explore the potential of using latent diffusion models, a
family of powerful generative models, for the task of reconstructing
naturalistic music from electroencephalogram (EEG) recordings. Unlike simpler
music with limited timbres, such as MIDI-generated tunes or monophonic pieces,
the focus here is on intricate music featuring a diverse array of instruments,
voices, and effects, rich in harmonics and timbre. This study represents an
initial foray into achieving general music reconstruction of high-quality using
non-invasive EEG data, employing an end-to-end training approach directly on
raw data without the need for manual pre-processing and channel selection. We
train our models on the public NMED-T dataset and perform quantitative
evaluation proposing neural embedding-based metrics. We additionally perform
song classification based on the generated tracks. Our work contributes to the
ongoing research in neural decoding and brain-computer interfaces, offering
insights into the feasibility of using EEG data for complex auditory
information reconstruction.
BEHAVIOR Vision Suite: Customizable Dataset Generation via Simulation
Authors: name: Yunhao Ge, name: Yihe Tang, name: Jiashu Xu, name: Cem Gokmen, name: Chengshu Li, name: Wensi Ai, name: Benjamin Jose Martinez, name: Arman Aydin, name: Mona Anvari, name: Ayush K Chakravarthy, name: Hong-Xing Yu, name: Josiah Wong, name: Sanjana Srivastava, name: Sharon Lee, name: Shengxin Zha, name: Laurent Itti, name: Yunzhu Li, name: Roberto Martín-Martín, name: Miao Liu, name: Pengchuan Zhang, name: Ruohan Zhang, name: Li Fei-Fei, name: Jiajun Wu
Published: 2024-05-15 17:57:56

The systematic evaluation and understanding of computer vision models under
varying conditions require large amounts of data with comprehensive and
customized labels, which real-world vision datasets rarely satisfy. While
current synthetic data generators offer a promising alternative, particularly
for embodied AI tasks, they often fall short for computer vision tasks due to
low asset and rendering quality, limited diversity, and unrealistic physical
properties. We introduce the BEHAVIOR Vision Suite (BVS), a set of tools and
assets to generate fully customized synthetic data for systematic evaluation of
computer vision models, based on the newly developed embodied AI benchmark,
BEHAVIOR-1K. BVS supports a large number of adjustable parameters at the scene
level (e.g., lighting, object placement), the object level (e.g., joint
configuration, attributes such as "filled" and "folded"), and the camera level
(e.g., field of view, focal length). Researchers can arbitrarily vary these
parameters during data generation to perform controlled experiments. We
showcase three example application scenarios: systematically evaluating the
robustness of models across different continuous axes of domain shift,
evaluating scene understanding models on the same set of images, and training
and evaluating simulation-to-real transfer for a novel vision task: unary and
binary state prediction. Project website:
https://behavior-vision-suite.github.io/