Orionnnn/hub-stats huggingface dataset:
PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
Authors: name: Soroush Nasiriany, name: Fei Xia, name: Wenhao Yu, name: Ted Xiao, name: Jacky Liang, name: Ishita Dasgupta, name: Annie Xie, name: Danny Driess, name: Ayzaan Wahid, name: Zhuo Xu, name: Quan Vuong, name: Tingnan Zhang, name: Tsang-Wei Edward Lee, name: Kuang-Huei Lee, name: Peng Xu, name: Sean Kirmani, name: Yuke Zhu, name: Andy Zeng, name: Karol Hausman, name: Nicolas Heess, name: Chelsea Finn, name: Sergey Levine, name: Brian Ichter
Published: 2024-02-12 18:33:47

Vision language models (VLMs) have shown impressive capabilities across a
variety of tasks, from logical reasoning to visual understanding. This opens
the door to richer interaction with the world, for example robotic control.
However, VLMs produce only textual outputs, while robotic control and other
spatial tasks require outputting continuous coordinates, actions, or
trajectories. How can we enable VLMs to handle such settings without
fine-tuning on task-specific data?
  In this paper, we propose a novel visual prompting approach for VLMs that we
call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as
iterative visual question answering. In each iteration, the image is annotated
with a visual representation of proposals that the VLM can refer to (e.g.,
candidate robot actions, localizations, or trajectories). The VLM then selects
the best ones for the task. These proposals are iteratively refined, allowing
the VLM to eventually zero in on the best available answer. We investigate
PIVOT on real-world robotic navigation, real-world manipulation from images,
instruction following in simulation, and additional spatial inference tasks
such as localization. We find, perhaps surprisingly, that our approach enables
zero-shot control of robotic systems without any robot training data,
navigation in a variety of environments, and other capabilities. Although
current performance is far from perfect, our work highlights potentials and
limitations of this new regime and shows a promising approach for
Internet-Scale VLMs in robotic and spatial reasoning domains. Website:
pivot-prompt.github.io and HuggingFace:
https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
Scaling Laws for Fine-Grained Mixture of Experts
Authors: name: Jakub Krajewski, name: Jan Ludziejewski, name: Kamil Adamczewski, name: Maciej Pióro, name: Michał Krutul, name: Szymon Antoniak, name: Kamil Ciebiera, name: Krystian Król, name: Tomasz Odrzygóźdź, name: Piotr Sankowski, name: Marek Cygan, name: Sebastian Jaszczur
Published: 2024-02-12 18:33:47

Mixture of Experts (MoE) models have emerged as a primary solution for
reducing the computational cost of Large Language Models. In this work, we
analyze their scaling properties, incorporating an expanded range of variables.
Specifically, we introduce a new hyperparameter, granularity, whose adjustment
enables precise control over the size of the experts. Building on this, we
establish scaling laws for fine-grained MoE, taking into account the number of
training tokens, model size, and granularity. Leveraging these laws, we derive
the optimal training configuration for a given computational budget. Our
findings not only show that MoE models consistently outperform dense
Transformers but also highlight that the efficiency gap between dense and MoE
models widens as we scale up the model size and training budget. Furthermore,
we demonstrate that the common practice of setting the size of experts in MoE
to mirror the feed-forward layer is not optimal at almost any computational
budget.
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned
  Language Models
Authors: name: Siddharth Karamcheti, name: Suraj Nair, name: Ashwin Balakrishna, name: Percy Liang, name: Thomas Kollar, name: Dorsa Sadigh
Published: 2024-02-12 18:21:14

Visually-conditioned language models (VLMs) have seen growing adoption in
applications such as visual dialogue, scene understanding, and robotic task
planning; adoption that has fueled a wealth of new models such as LLaVa,
InstructBLIP, and PaLI-3. Despite the volume of new releases, key design
decisions around image preprocessing, architecture, and optimization are
under-explored, making it challenging to understand what factors account for
model performance - a challenge further complicated by the lack of objective,
consistent evaluations. To address these gaps, we first compile a suite of
standardized evaluations spanning visual question answering, object
localization from language, and targeted challenge sets that probe properties
such as hallucination; evaluations that provide calibrated, fine-grained
insight into a VLM's capabilities. Second, we rigorously investigate VLMs along
key design axes, including pretrained visual representations and quantifying
the tradeoffs of using base vs. instruct-tuned language models, amongst others.
We couple our analysis with three resource contributions: (1) a unified
framework for evaluating VLMs, (2) optimized, flexible code for VLM training,
and (3) checkpoints for all models, including a family of VLMs at the 7-13B
scale that strictly outperform InstructBLIP and LLaVa v1.5, the
state-of-the-art in open-source VLMs.
Aya Model: An Instruction Finetuned Open-Access Multilingual Language
  Model
Authors: name: Ahmet Üstün, name: Viraat Aryabumi, name: Zheng-Xin Yong, name: Wei-Yin Ko, name: Daniel D'souza, name: Gbemileke Onilude, name: Neel Bhandari, name: Shivalika Singh, name: Hui-Lee Ooi, name: Amr Kayid, name: Freddie Vargus, name: Phil Blunsom, name: Shayne Longpre, name: Niklas Muennighoff, name: Marzieh Fadaee, name: Julia Kreutzer, name: Sara Hooker
Published: 2024-02-12 17:34:13

Recent breakthroughs in large language models (LLMs) have centered around a
handful of data-rich languages. What does it take to broaden access to
breakthroughs beyond first-class citizen languages? Our work introduces Aya, a
massively multilingual generative language model that follows instructions in
101 languages of which over 50% are considered as lower-resourced. Aya
outperforms mT0 and BLOOMZ on the majority of tasks while covering double the
number of languages. We introduce extensive new evaluation suites that broaden
the state-of-art for multilingual eval across 99 languages -- including
discriminative and generative tasks, human evaluation, and simulated win rates
that cover both held-out tasks and in-distribution performance. Furthermore, we
conduct detailed investigations on the optimal finetuning mixture composition,
data pruning, as well as the toxicity, bias, and safety of our models. We
open-source our instruction datasets and our model at
https://hf.co/CohereForAI/aya-101
AutoMathText: Autonomous Data Selection with Language Models for
  Mathematical Texts
Authors: name: Yifan Zhang, name: Yifan Luo, name: Yang Yuan, name: Andrew Chi-Chih Yao
Published: 2024-02-12 13:09:21

To improve language models' proficiency in mathematical reasoning via
continual pretraining, we introduce a novel strategy that leverages base
language models for autonomous data selection. Departing from conventional
supervised fine-tuning or trained classifiers with human-annotated data, our
approach utilizes meta-prompted language models as zero-shot verifiers to
autonomously evaluate and select high-quality mathematical content, and we
release the curated open-source AutoMathText dataset encompassing over 200GB of
data. To demonstrate the efficacy of our method, we continuously pretrained a
7B-parameter Mistral language model on the AutoMathText dataset, achieving
substantial improvements in downstream performance on the MATH dataset with a
token amount reduced by orders of magnitude compared to previous continuous
pretraining works. Our method showcases a 2 times increase in pretraining token
efficiency compared to baselines, underscoring the potential of our approach in
enhancing models' mathematical reasoning capabilities. The AutoMathText dataset
is available at https://huggingface.co/datasets/math-ai/AutoMathText. The code
is available at https://github.com/yifanzhang-pro/AutoMathText.
Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Authors: name: Haoyu Wang, name: Guozheng Ma, name: Ziqiao Meng, name: Zeyu Qin, name: Li Shen, name: Zhong Zhang, name: Bingzhe Wu, name: Liu Liu, name: Yatao Bian, name: Tingyang Xu, name: Xueqian Wang, name: Peilin Zhao
Published: 2024-02-12 12:30:42

Self-alignment is an effective way to reduce the cost of human annotation
while ensuring promising model capability. However, most current methods
complete the data collection and training steps in a single round, which may
overlook the continuously improving ability of self-aligned models. This gives
rise to a key query: What if we do multi-time bootstrapping self-alignment?
Does this strategy enhance model performance or lead to rapid degradation? In
this paper, our pioneering exploration delves into the impact of bootstrapping
self-alignment on large language models. Our findings reveal that bootstrapping
self-alignment markedly surpasses the single-round approach, by guaranteeing
data diversity from in-context learning. To further exploit the capabilities of
bootstrapping, we investigate and adjust the training order of data, which
yields improved performance of the model. Drawing on these findings, we propose
Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
training recipe, we propose SOFT+ which further boost self-alignment's
performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
various classification and generation tasks, highlighting the potential of
bootstrapping self-alignment on continually enhancing model alignment
performance.
OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Authors: name: Zhiyong Wu, name: Chengcheng Han, name: Zichen Ding, name: Zhenmin Weng, name: Zhoumianze Liu, name: Shunyu Yao, name: Tao Yu, name: Lingpeng Kong
Published: 2024-02-12 07:29:22

Autonomous interaction with the computer has been a longstanding challenge
with great potential, and the recent proliferation of large language models
(LLMs) has markedly accelerated progress in building digital agents. However,
most of these agents are designed to interact with a narrow domain, such as a
specific software or website. This narrow focus constrains their applicability
for general computer tasks. To this end, we introduce OS-Copilot, a framework
to build generalist agents capable of interfacing with comprehensive elements
in an operating system (OS), including the web, code terminals, files,
multimedia, and various third-party applications. We use OS-Copilot to create
FRIDAY, a self-improving embodied agent for automating general computer tasks.
On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods
by 35%, showcasing strong generalization to unseen applications via accumulated
skills from previous tasks. We also present numerical and quantitative evidence
that FRIDAY learns to control and self-improve on Excel and Powerpoint with
minimal supervision. Our OS-Copilot framework and empirical findings provide
infrastructure and insights for future research toward more capable and
general-purpose computer agents.
GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided
  Generative Gaussian Splatting
Authors: name: Xiaoyu Zhou, name: Xingjian Ran, name: Yajiao Xiong, name: Jinlin He, name: Zhiwei Lin, name: Yongtao Wang, name: Deqing Sun, name: Ming-Hsuan Yang
Published: 2024-02-11 13:40:08

We present GALA3D, generative 3D GAussians with LAyout-guided control, for
effective compositional text-to-3D generation. We first utilize large language
models (LLMs) to generate the initial layout and introduce a layout-guided 3D
Gaussian representation for 3D content generation with adaptive geometric
constraints. We then propose an object-scene compositional optimization
mechanism with conditioned diffusion to collaboratively generate realistic 3D
scenes with consistent geometry, texture, scale, and accurate interactions
among multiple objects while simultaneously adjusting the coarse layout priors
extracted from the LLMs to align with the generated scene. Experiments show
that GALA3D is a user-friendly, end-to-end framework for state-of-the-art
scene-level 3D content generation and controllable editing while ensuring the
high fidelity of object-level entities within the scene. Source codes and
models will be available at https://gala3d.github.io/.
A Tale of Tails: Model Collapse as a Change of Scaling Laws
Authors: name: Elvis Dohmatob, name: Yunzhen Feng, name: Pu Yang, name: Francois Charton, name: Julia Kempe
Published: 2024-02-10 21:06:34

As AI model size grows, neural scaling laws have become a crucial tool to
predict the improvements of large models when increasing capacity and the size
of original (human or natural) training data. Yet, the widespread use of
popular models means that the ecosystem of online data and text will co-evolve
to progressively contain increased amounts of synthesized data. In this paper
we ask: How will the scaling laws change in the inevitable regime where
synthetic data makes its way into the training corpus? Will future models,
still improve, or be doomed to degenerate up to total (model) collapse? We
develop a theoretical framework of model collapse through the lens of scaling
laws. We discover a wide range of decay phenomena, analyzing loss of scaling,
shifted scaling with number of generations, the ''un-learning" of skills, and
grokking when mixing human and synthesized data. Our theory is validated by
large-scale experiments with a transformer on an arithmetic task and text
generation using the large language model Llama2.
Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts
  Models
Authors: name: Keisuke Kamahori, name: Yile Gu, name: Kan Zhu, name: Baris Kasikci
Published: 2024-02-10 19:54:08

Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture
are showing promising performance on various tasks. However, running them on
resource-constrained settings, where GPU memory resources are not abundant, is
challenging due to huge model sizes. Existing systems that offload model
weights to CPU memory suffer from the significant overhead of frequently moving
data between CPU and GPU. In this paper, we propose Fiddler, a
resource-efficient inference engine with CPU-GPU orchestration for MoE models.
The key idea of Fiddler is to use the computation ability of the CPU to
minimize the data movement between the CPU and GPU. Our evaluation shows that
Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in
parameters, to generate over 3 tokens per second on a single GPU with 24GB
memory, showing an order of magnitude improvement over existing methods. The
code of Fiddler is publicly available at
https://github.com/efeslab/fiddler
ChemLLM: A Chemical Large Language Model
Authors: name: Di Zhang, name: Wei Liu, name: Qian Tan, name: Jingdan Chen, name: Hang Yan, name: Yuliang Yan, name: Jiatong Li, name: Weiran Huang, name: Xiangyu Yue, name: Dongzhan Zhou, name: Shufei Zhang, name: Mao Su, name: Hansen Zhong, name: Yuqiang Li, name: Wanli Ouyang
Published: 2024-02-10 01:11:59

Large language models (LLMs) have made impressive progress in chemistry
applications, including molecular property prediction, molecular generation,
experimental protocol design, etc. However, the community lacks a
dialogue-based model specifically designed for chemistry. The challenge arises
from the fact that most chemical data and scientific knowledge are primarily
stored in structured databases, and the direct use of these structured data
compromises the model's ability to maintain coherent dialogue. To tackle this
issue, we develop a novel template-based instruction construction method that
transforms structured knowledge into plain dialogue, making it suitable for
language model training. By leveraging this approach, we develop ChemLLM, the
first large language model dedicated to chemistry, capable of performing
various tasks across chemical disciplines with smooth dialogue interaction.
ChemLLM beats GPT-3.5 on all three principal tasks in chemistry, i.e., name
conversion, molecular caption, and reaction prediction, and surpasses GPT-4 on
two of them. Remarkably, ChemLLM also shows exceptional adaptability to related
mathematical and physical tasks despite being trained mainly on
chemical-centric corpora. Furthermore, ChemLLM demonstrates proficiency in
specialized NLP tasks within chemistry, such as literature translation and
cheminformatic programming. ChemLLM opens up a new avenue for exploration
within chemical studies, while our method of integrating structured chemical
knowledge into dialogue systems sets a new frontier for developing LLMs across
various scientific fields. Codes, Datasets, and Model weights are publicly
accessible at hf.co/AI4Chem/ChemLLM-7B-Chat.
Suppressing Pink Elephants with Direct Principle Feedback
Authors: name: Louis Castricato, name: Nathan Lile, name: Suraj Anand, name: Hailey Schoelkopf, name: Siddharth Verma, name: Stella Biderman
Published: 2024-02-12 18:57:46

Existing methods for controlling language models, such as RLHF and
Constitutional AI, involve determining which LLM behaviors are desirable and
training them into a language model. However, in many cases, it is desirable
for LLMs to be controllable at inference time, so that they can be
used in multiple contexts with diverse needs. We illustrate this with the
Pink Elephant Problem: instructing an LLM to avoid discussing a
certain entity (a ``Pink Elephant''), and instead discuss a preferred entity
(``Grey Elephant''). We apply a novel simplification of Constitutional AI,
Direct Principle Feedback, which skips the ranking of responses and
uses DPO directly on critiques and revisions. Our results show that after DPF
fine-tuning on our synthetic Pink Elephants dataset, our 13B fine-tuned LLaMA 2
model significantly outperforms Llama-2-13B-Chat and a prompted baseline, and
performs as well as GPT-4 in on our curated test set assessing the Pink
Elephant Problem.
InternLM-Math: Open Math Large Language Models Toward Verifiable
  Reasoning
Authors: name: Huaiyuan Ying, name: Shuo Zhang, name: Linyang Li, name: Zhejian Zhou, name: Yunfan Shao, name: Zhaoye Fei, name: Yichuan Ma, name: Jiawei Hong, name: Kuikun Liu, name: Ziyi Wang, name: Yudong Wang, name: Zijian Wu, name: Shuaibin Li, name: Fengzhe Zhou, name: Hongwei Liu, name: Songyang Zhang, name: Wenwei Zhang, name: Hang Yan, name: Xipeng Qiu, name: Jiayu Wang, name: Kai Chen, name: Dahua Lin
Published: 2024-02-09 11:22:08

The math abilities of large language models can represent their abstract
reasoning ability. In this paper, we introduce and open-source our math
reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We
unify chain-of-thought reasoning, reward modeling, formal reasoning, data
augmentation, and code interpreter in a unified seq2seq format and supervise
our model to be a versatile math reasoner, verifier, prover, and augmenter.
These abilities can be used to develop the next math LLMs or self-iteration.
InternLM-Math obtains open-sourced state-of-the-art performance under the
setting of in-context learning, supervised fine-tuning, and code-assisted
reasoning in various informal and formal benchmarks including GSM8K, MATH,
Hungary math exam, MathBench-ZH, and MiniF2F. Our pre-trained model achieves
30.3 on the MiniF2F test set without fine-tuning. We further explore how to use
LEAN to solve math problems and study its performance under the setting of
multi-task learning which shows the possibility of using LEAN as a unified
platform for solving and proving in math. Our models, codes, and data are
released at https://github.com/InternLM/InternLM-Math.
Keyframer: Empowering Animation Design using Large Language Models
Authors: name: Tiffany Tseng, name: Ruijia Cheng, name: Jeffrey Nichols
Published: 2024-02-08 21:43:30

Large language models (LLMs) have the potential to impact a wide range of
creative domains, but the application of LLMs to animation is underexplored and
presents novel challenges such as how users might effectively describe motion
in natural language. In this paper, we present Keyframer, a design tool for
animating static images (SVGs) with natural language. Informed by interviews
with professional animation designers and engineers, Keyframer supports
exploration and refinement of animations through the combination of prompting
and direct editing of generated output. The system also enables users to
request design variants, supporting comparison and ideation. Through a user
study with 13 participants, we contribute a characterization of user prompting
strategies, including a taxonomy of semantic prompt types for describing motion
and a 'decomposed' prompting style where users continually adapt their goals in
response to generated output.We share how direct editing along with prompting
enables iteration beyond one-shot prompting interfaces common in generative
tools today. Through this work, we propose how LLMs might empower a range of
audiences to engage with animation creation.
SubGen: Token Generation in Sublinear Time and Memory
Authors: name: Amir Zandieh, name: Insu Han, name: Vahab Mirrokni, name: Amin Karbasi
Published: 2024-02-08 22:17:40

Despite the significant success of large language models (LLMs), their
extensive memory requirements pose challenges for deploying them in
long-context token generation. The substantial memory footprint of LLM decoders
arises from the necessity to store all previous tokens in the attention module,
a requirement imposed by key-value (KV) caching. In this work, our focus is on
developing an efficient compression technique for the KV cache. Empirical
evidence indicates a significant clustering tendency within key embeddings in
the attention module. Building on this key insight, we have devised a novel
caching method with sublinear complexity, employing online clustering on key
tokens and online ell_2 sampling on values. The result is a provably
accurate and efficient attention decoding algorithm, termed SubGen. Not only
does this algorithm ensure a sublinear memory footprint and sublinear time
complexity, but we also establish a tight error bound for our approach.
Empirical evaluations on long-context question-answering tasks demonstrate that
SubGen significantly outperforms existing and state-of-the-art KV cache
compression methods in terms of performance and efficiency.
ViGoR: Improving Visual Grounding of Large Vision Language Models with
  Fine-Grained Reward Modeling
Authors: name: Siming Yan, name: Min Bai, name: Weifeng Chen, name: Xiong Zhou, name: Qixing Huang, name: Li Erran Li
Published: 2024-02-09 01:00:14

By combining natural language understanding and the generation capabilities
and breadth of knowledge of large language models with image perception, recent
large vision language models (LVLMs) have shown unprecedented reasoning
capabilities in the real world. However, the generated text often suffers from
inaccurate grounding in the visual input, resulting in errors such as
hallucinating nonexistent scene elements, missing significant parts of the
scene, and inferring incorrect attributes and relationships between objects. To
address these issues, we introduce a novel framework, ViGoR (Visual Grounding
Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
modeling to significantly enhance the visual grounding of LVLMs over
pre-trained baselines. This improvement is efficiently achieved using much
cheaper human evaluations instead of full supervisions, as well as automated
methods. We show the effectiveness of our approach through numerous metrics on
several benchmarks. Additionally, we construct a comprehensive and challenging
dataset specifically designed to validate the visual grounding capabilities of
LVLMs. Finally, we plan to release our human annotation comprising
approximately 16,000 images and generated text pairs with fine-grained
evaluations to contribute to related research in the community.