Orionnnn/hub-stats huggingface dataset:
Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate
  Reward Hacking
Authors: name: Jacob Eisenstein, name: Chirag Nagpal, name: Alekh Agarwal, name: Ahmad Beirami, name: Alex D'Amour, name: DJ Dvijotham, name: Adam Fisch, name: Katherine Heller, name: Stephen Pfohl, name: Deepak Ramachandran, name: Peter Shaw, name: Jonathan Berant
Published: 2023-12-14 18:59:04

Reward models play a key role in aligning language model applications towards
human preferences. However, this setup creates an incentive for the language
model to exploit errors in the reward model to achieve high estimated reward, a
phenomenon often termed reward hacking. A natural mitigation is to train
an ensemble of reward models, aggregating over model outputs to obtain a more
robust reward estimate. We explore the application of reward ensembles to
alignment at both training time (through reinforcement learning) and inference
time (through reranking). First, we show that reward models are
underspecified: reward models that perform similarly in-distribution can
yield very different rewards when used in alignment, due to distribution shift.
Second, underspecification results in overoptimization, where alignment to one
reward model does not improve reward as measured by another reward model
trained on the same data. Third, overoptimization is mitigated by the use of
reward ensembles, and ensembles that vary by their pretraining seeds
lead to better generalization than ensembles that differ only by their
fine-tuning seeds, with both outperforming individual reward models.
However, even pretrain reward ensembles do not eliminate reward hacking: we
show several qualitative reward hacking phenomena that are not mitigated by
ensembling because all reward models in the ensemble exhibit similar error
patterns.
TinyGSM: achieving >80% on GSM8k with small language models
Authors: name: Bingbin Liu, name: Sebastien Bubeck, name: Ronen Eldan, name: Janardhan Kulkarni, name: Yuanzhi Li, name: Anh Nguyen, name: Rachel Ward, name: Yi Zhang
Published: 2023-12-14 18:58:28

Small-scale models offer various computational advantages, and yet to which
extent size is critical for problem-solving abilities remains an open question.
Specifically for solving grade school math, the smallest model size so far
required to break the 80\% barrier on the GSM8K benchmark remains to be 34B.
Our work studies how high-quality datasets may be the key for small language
models to acquire mathematical reasoning. We introduce TinyGSM, a
synthetic dataset of 12.3M grade school math problems paired with Python
solutions, generated fully by GPT-3.5. After finetuning on TinyGSM, we
find that a duo of a 1.3B generation model and a 1.3B verifier model can
achieve 81.5\% accuracy, outperforming existing models that are orders of
magnitude larger. This also rivals the performance of the GPT-3.5 ``teacher''
model (77.4\%), from which our model's training data is generated. Our approach
is simple and has two key components: 1) the high-quality dataset
TinyGSM, 2) the use of a verifier, which selects the final outputs
from multiple candidate generations.
Pixel Aligned Language Models
Authors: name: Jiarui Xu, name: Xingyi Zhou, name: Shen Yan, name: Xiuye Gu, name: Anurag Arnab, name: Chen Sun, name: Xiaolong Wang, name: Cordelia Schmid
Published: 2023-12-14 18:57:58

Large language models have achieved great success in recent years, so as
their variants in vision. Existing vision-language models can describe images
in natural languages, answer visual-related questions, or perform complex
reasoning about the image. However, it is yet unclear how localization tasks,
such as word grounding or referring localization, can be performed using large
language models. In this work, we aim to develop a vision-language model that
can take locations, for example, a set of points or boxes, as either inputs or
outputs. When taking locations as inputs, the model performs
location-conditioned captioning, which generates captions for the indicated
object or region. When generating locations as outputs, our model regresses
pixel coordinates for each output word generated by the language model, and
thus performs dense word grounding. Our model is pre-trained on the Localized
Narrative dataset, which contains pixel-word-aligned captioning from human
attention. We show our model can be applied to various location-aware
vision-language tasks, including referring localization, location-conditioned
captioning, and dense object captioning, archiving state-of-the-art performance
on RefCOCO and Visual Genome. Project page: https://jerryxu.net/PixelLLM .
Vision-Language Models as a Source of Rewards
Authors: name: Kate Baumli, name: Satinder Baveja, name: Feryal Behbahani, name: Harris Chan, name: Gheorghe Comanici, name: Sebastian Flennerhag, name: Maxime Gazeau, name: Kristian Holsheimer, name: Dan Horgan, name: Michael Laskin, name: Clare Lyle, name: Hussain Masoom, name: Kay McKinney, name: Volodymyr Mnih, name: Alexander Neitz, name: Fabio Pardo, name: Jack Parker-Holder, name: John Quan, name: Tim Rocktäschel, name: Himanshu Sahni, name: Tom Schaul, name: Yannick Schroecker, name: Stephen Spencer, name: Richie Steigerwald, name: Luyu Wang, name: Lei Zhang
Published: 2023-12-14 18:06:17

Building generalist agents that can accomplish many goals in rich open-ended
environments is one of the research frontiers for reinforcement learning. A key
limiting factor for building generalist agents with RL has been the need for a
large number of reward functions for achieving different goals. We investigate
the feasibility of using off-the-shelf vision-language models, or VLMs, as
sources of rewards for reinforcement learning agents. We show how rewards for
visual achievement of a variety of language goals can be derived from the CLIP
family of models, and used to train RL agents that can achieve a variety of
language goals. We showcase this approach in two distinct visual domains and
present a scaling trend showing how larger VLMs lead to more accurate rewards
for visual goal achievement, which in turn produces more capable RL agents.
General Object Foundation Model for Images and Videos at Scale
Authors: name: Junfeng Wu, name: Yi Jiang, name: Qihao Liu, name: Zehuan Yuan, name: Xiang Bai, name: Song Bai
Published: 2023-12-14 17:26:00

We present GLEE in this work, an object-level foundation model for locating
and identifying objects in images and videos. Through a unified framework, GLEE
accomplishes detection, segmentation, tracking, grounding, and identification
of arbitrary objects in the open world scenario for various object perception
tasks. Adopting a cohesive learning strategy, GLEE acquires knowledge from
diverse data sources with varying supervision levels to formulate general
object representations, excelling in zero-shot transfer to new data and tasks.
Specifically, we employ an image encoder, text encoder, and visual prompter to
handle multi-modal inputs, enabling to simultaneously solve various
object-centric downstream tasks while maintaining state-of-the-art performance.
Demonstrated through extensive training on over five million images from
diverse benchmarks, GLEE exhibits remarkable versatility and improved
generalization performance, efficiently tackling downstream tasks without the
need for task-specific adaptation. By integrating large volumes of
automatically labeled data, we further enhance its zero-shot generalization
capabilities. Additionally, GLEE is capable of being integrated into Large
Language Models, serving as a foundational model to provide universal
object-level information for multi-modal tasks. We hope that the versatility
and universality of our method will mark a significant step in the development
of efficient visual foundation models for AGI systems. The model and code will
be released at https://glee-vision.github.io .
CogAgent: A Visual Language Model for GUI Agents
Authors: name: Wenyi Hong, name: Weihan Wang, name: Qingsong Lv, name: Jiazheng Xu, name: Wenmeng Yu, name: Junhui Ji, name: Yan Wang, name: Zihan Wang, name: Yuxiao Dong, name: Ming Ding, name: Jie Tang
Published: 2023-12-14 13:20:57

People are spending an enormous amount of time on digital devices through
graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
language models (LLMs) such as ChatGPT can assist people in tasks like writing
emails, but struggle to understand and interact with GUIs, thus limiting their
potential to increase automation levels. In this paper, we introduce CogAgent,
an 18-billion-parameter visual language model (VLM) specializing in GUI
understanding and navigation. By utilizing both low-resolution and
high-resolution image encoders, CogAgent supports input at a resolution of
1120*1120, enabling it to recognize tiny page elements and text. As a
generalist visual language model, CogAgent achieves the state of the art on
five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
only screenshots as input, outperforms LLM-based methods that consume extracted
HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
advancing the state of the art. The model and codes are available at
https://github.com/THUDM/CogVLM.
TigerBot: An Open Multilingual Multitask LLM
Authors: name: Ye Chen, name: Wei Cai, name: Liangmin Wu, name: Xiaowei Li, name: Zhanxuan Xin, name: Cong Fu
Published: 2023-12-14 07:05:42

We release and introduce the TigerBot family of large language models (LLMs),
consisting of base and chat models, sized from 7, 13, 70 and 180 billion
parameters. We develop our models embarking from Llama-2 and BLOOM, and push
the boundary further in data, training algorithm, infrastructure, and
application tools. Our models yield meaningful performance gain over SOTA
open-source models, e.g., Llama-2, specifically 6\% gain in English and 20\%
gain in Chinese. TigerBot model family also achieves leading performance in
major academic and industrial benchmarks and leaderboards. We believe that
TigerBot represents just a snapshot of lightning-fast progression in LLM
open-source community. Therefore, we are thrilled to give back by publicly
releasing our models and reporting our approach behind, with additional
emphases on building SOTA LLMs in a democratized way and making LLMs of use in
real-world applications.
Zebra: Extending Context Window with Layerwise Grouped Local-Global
  Attention
Authors: name: Kaiqiang Song, name: Xiaoyang Wang, name: Sangwoo Cho, name: Xiaoman Pan, name: Dong Yu
Published: 2023-12-14 02:45:31

This paper introduces a novel approach to enhance the capabilities of Large
Language Models (LLMs) in processing and understanding extensive text
sequences, a critical aspect in applications requiring deep comprehension and
synthesis of large volumes of information. Recognizing the inherent challenges
in extending the context window for LLMs, primarily built on Transformer
architecture, we propose a new model architecture, referred to as Zebra. This
architecture efficiently manages the quadratic time and memory complexity
issues associated with full attention in the Transformer by employing grouped
local-global attention layers. Our model, akin to a zebra's alternating
stripes, balances local and global attention layers, significantly reducing
computational requirements and memory consumption. Comprehensive experiments,
including pretraining from scratch, continuation of long context adaptation
training, and long instruction tuning, are conducted to evaluate the Zebra's
performance. The results show that Zebra achieves comparable or superior
performance on both short and long sequence benchmarks, while also enhancing
training and inference efficiency.
ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric
  Strategy for Diverse Generative Tasks
Authors: name: Xiaoxia Wu, name: Haojun Xia, name: Stephen Youn, name: Zhen Zheng, name: Shiyang Chen, name: Arash Bakhtiari, name: Michael Wyatt, name: Yuxiong He, name: Olatunji Ruwase, name: Leon Song, name: Zhewei Yao
Published: 2023-12-14 01:06:37

This study examines 4-bit quantization methods like GPTQ in large language
models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
extend task scope to more generative categories such as code generation and
abstractive summarization, in which we found that INT4 quantization can
significantly underperform. However, simply shifting to higher precision
formats like FP6 has been particularly challenging, thus overlooked, due to
poor performance caused by the lack of sophisticated integration and system
acceleration strategies on current AI hardware. Our results show that FP6, even
with a coarse-grain quantization scheme, performs robustly across various
algorithms and tasks, demonstrating its superiority in accuracy and
versatility. Notably, with the FP6 quantization, \codestar-15B model performs
comparably to its FP16 counterpart in code generation, and for smaller models
like the 406M it closely matches their baselines in summarization. Neither can
be achieved by INT4. To better accommodate various AI hardware and achieve the
best system performance, we propose a novel 4+2 design for FP6 to achieve
similar latency to the state-of-the-art INT4 fine-grain quantization. With our
design, FP6 can become a promising solution to the current 4-bit quantization
methods used in LLMs.
A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style
  Models on Dense Captions
Authors: name: Jack Urbanek, name: Florian Bordes, name: Pietro Astolfi, name: Mary Williamson, name: Vasu Sharma, name: Adriana Romero-Soriano
Published: 2023-12-14 00:42:23

Curation methods for massive vision-language datasets trade off between
dataset size and quality. However, even the highest quality of available
curated captions are far too short to capture the rich visual detail in an
image. To show the value of dense and highly-aligned image-text pairs, we
collect the Densely Captioned Images (DCI) dataset, containing 8012 natural
images human-annotated with mask-aligned descriptions averaging above 1000
words each. With precise and reliable captions associated with specific parts
of an image, we can evaluate vision-language models' (VLMs) understanding of
image content with a novel task that matches each caption with its
corresponding subcrop. As current models are often limited to 77 text tokens,
we also introduce a summarized version (sDCI) in which each caption length is
limited. We show that modern techniques that make progress on standard
benchmarks do not correspond with significant improvement on our sDCI based
benchmark. Lastly, we finetune CLIP using sDCI and show significant
improvements over the baseline despite a small training set. By releasing the
first human annotated dense image captioning dataset, we hope to enable the
development of new benchmarks or fine-tuning recipes for the next generation of
VLMs to come.
Holodeck: Language Guided Generation of 3D Embodied AI Environments
Authors: name: Yue Yang, name: Fan-Yun Sun, name: Luca Weihs, name: Eli VanderBilt, name: Alvaro Herrasti, name: Winson Han, name: Jiajun Wu, name: Nick Haber, name: Ranjay Krishna, name: Lingjie Liu, name: Chris Callison-Burch, name: Mark Yatskar, name: Aniruddha Kembhavi, name: Christopher Clark
Published: 2023-12-14 16:04:14

3D simulated environments play a critical role in Embodied AI, but their
creation requires expertise and extensive manual effort, restricting their
diversity and scope. To mitigate this limitation, we present Holodeck, a system
that generates 3D environments to match a user-supplied prompt fully
automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
museums, adjust the designs for styles, and can capture the semantics of
complex queries such as "apartment for a researcher with a cat" and "office of
a professor who is a fan of Star Wars". Holodeck leverages a large language
model (GPT-4) for common sense knowledge about what the scene might look like
and uses a large collection of 3D assets from Objaverse to populate the scene
with diverse objects. To address the challenge of positioning objects
correctly, we prompt GPT-4 to generate spatial relational constraints between
objects and then optimize the layout to satisfy those constraints. Our
large-scale human evaluation shows that annotators prefer Holodeck over
manually designed procedural baselines in residential scenes and that Holodeck
can produce high-quality outputs for diverse scene types. We also demonstrate
an exciting application of Holodeck in Embodied AI, training agents to navigate
in novel scenes like music rooms and daycares without human-constructed data,
which is a significant step forward in developing general-purpose embodied
agents.
Modeling Complex Mathematical Reasoning via Large Language Model based
  MathAgent
Authors: name: Haoran Liao, name: Qinyi Du, name: Shaohua Hu, name: Hao He, name: Yanyan Xu, name: Jidong Tian, name: Yaohui Jin
Published: 2023-12-14 13:33:50

Large language models (LLMs) face challenges in solving complex mathematical
problems that require comprehensive capacities to parse the statements,
associate domain knowledge, perform compound logical reasoning, and integrate
the intermediate rationales. Tackling all these problems once could be arduous
for LLMs, thus leading to confusion in generation. In this work, we explore the
potential of enhancing LLMs with agents by meticulous decomposition and
modeling of mathematical reasoning process. Specifically, we propose a formal
description of the mathematical solving and extend LLMs with an agent-based
zero-shot framework named
Planner-Reasoner-Executor-Reflector (PRER). We
further provide and implement two MathAgents that define the logical forms and
inherent relations via a pool of actions in different grains and orientations:
MathAgent-M adapts its actions to LLMs, while MathAgent-H aligns with
humankind. Experiments on miniF2F and MATH have demonstrated the effectiveness
of PRER and proposed MathAgents, achieving an increase of
12.3%(53.9%66.2%) on the MiniF2F, 9.2%
(49.8%59.0%) on MATH, and
13.2%(23.2%35.4%) for level-5 problems of MATH against
GPT-4. Further analytical results provide more insightful perspectives on
exploiting the behaviors of LLMs as agents.
Invariant Graph Transformer
Authors: name: Zhe Xu, name: Menghai Pan, name: Yuzhong Chen, name: Huiyuan Chen, name: Yuchen Yan, name: Mahashweta Das, name: Hanghang Tong
Published: 2023-12-13 02:56:26

Rationale discovery is defined as finding a subset of the input data that
maximally supports the prediction of downstream tasks. In graph machine
learning context, graph rationale is defined to locate the critical subgraph in
the given graph topology, which fundamentally determines the prediction
results. In contrast to the rationale subgraph, the remaining subgraph is named
the environment subgraph. Graph rationalization can enhance the model
performance as the mapping between the graph rationale and prediction label is
viewed as invariant, by assumption. To ensure the discriminative power of the
extracted rationale subgraphs, a key technique named "intervention" is applied.
The core idea of intervention is that given any changing environment subgraphs,
the semantics from the rationale subgraph is invariant, which guarantees the
correct prediction result. However, most, if not all, of the existing
rationalization works on graph data develop their intervention strategies on
the graph level, which is coarse-grained. In this paper, we propose
well-tailored intervention strategies on graph data. Our idea is driven by the
development of Transformer models, whose self-attention module provides rich
interactions between input nodes. Based on the self-attention module, our
proposed invariant graph Transformer (IGT) can achieve fine-grained, more
specifically, node-level and virtual node-level intervention. Our comprehensive
experiments involve 7 real-world datasets, and the proposed IGT shows
significant performance advantages compared to 13 baseline methods.
ProNeRF: Learning Efficient Projection-Aware Ray Sampling for
  Fine-Grained Implicit Neural Radiance Fields
Authors: name: Juan Luis Gonzalez Bello, name: Minh-Quan Viet Bui, name: Munchurl Kim
Published: 2023-12-13 13:37:32

Recent advances in neural rendering have shown that, albeit slow, implicit
compact models can learn a scene's geometries and view-dependent appearances
from multiple views. To maintain such a small memory footprint but achieve
faster inference times, recent works have adopted `sampler' networks that
adaptively sample a small subset of points along each ray in the implicit
neural radiance fields. Although these methods achieve up to a 10times
reduction in rendering time, they still suffer from considerable quality
degradation compared to the vanilla NeRF. In contrast, we propose ProNeRF,
which provides an optimal trade-off between memory footprint (similar to NeRF),
speed (faster than HyperReel), and quality (better than K-Planes). ProNeRF is
equipped with a novel projection-aware sampling (PAS) network together with a
new training strategy for ray exploration and exploitation, allowing for
efficient fine-grained particle sampling. Our ProNeRF yields state-of-the-art
metrics, being 15-23x faster with 0.65dB higher PSNR than NeRF and yielding
0.95dB higher PSNR than the best published sampler-based method, HyperReel. Our
exploration and exploitation training strategy allows ProNeRF to learn the full
scenes' color and density distributions while also learning efficient ray
sampling focused on the highest-density regions. We provide extensive
experimental results that support the effectiveness of our method on the widely
adopted forward-facing and 360 datasets, LLFF and Blender, respectively.
CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor
Authors: name: Shuyang Sun, name: Runjia Li, name: Philip Torr, name: Xiuye Gu, name: Siyang Li
Published: 2023-12-12 19:00:04

Existing open-vocabulary image segmentation methods require a fine-tuning
step on mask annotations and/or image-text datasets. Mask labels are
labor-intensive, which limits the number of categories in segmentation
datasets. As a result, the open-vocabulary capacity of pre-trained VLMs is
severely reduced after fine-tuning. However, without fine-tuning, VLMs trained
under weak image-text supervision tend to make suboptimal mask predictions when
there are text queries referring to non-existing concepts in the image. To
alleviate these issues, we introduce a novel recurrent framework that
progressively filters out irrelevant texts and enhances mask quality without
training efforts. The recurrent unit is a two-stage segmenter built upon a VLM
with frozen weights. Thus, our model retains the VLM's broad vocabulary space
and strengthens its segmentation capability. Experimental results show that our
method outperforms not only the training-free counterparts, but also those
fine-tuned with millions of additional data samples, and sets new
state-of-the-art records for both zero-shot semantic and referring image
segmentation tasks. Specifically, we improve the current record by 28.8, 16.0,
and 6.9 mIoU on Pascal VOC, COCO Object, and Pascal Context.