Orionnnn/hub-stats huggingface dataset:
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Authors: name: Jianwei Yang, name: Hao Zhang, name: Feng Li, name: Xueyan Zou, name: Chunyuan Li, name: Jianfeng Gao
Published: 2023-10-17 17:51:31

We present Set-of-Mark (SoM), a new visual prompting method, to unleash the
visual grounding abilities of large multimodal models (LMMs), such as GPT-4V.
As illustrated in Fig. 1 (right), we employ off-the-shelf interactive
segmentation models, such as SAM, to partition an image into regions at
different levels of granularity, and overlay these regions with a set of marks
e.g., alphanumerics, masks, boxes. Using the marked image as input, GPT-4V can
answer the questions that require visual grounding. We perform a comprehensive
empirical study to validate the effectiveness of SoM on a wide range of
fine-grained vision and multimodal tasks. For example, our experiments show
that GPT-4V with SoM outperforms the state-of-the-art fully-finetuned referring
segmentation model on RefCOCOg in a zero-shot setting.
Interactive Task Planning with Language Models
Authors: name: Boyi Li, name: Philipp Wu, name: Pieter Abbeel, name: Jitendra Malik
Published: 2023-10-16 17:59:12

An interactive robot framework accomplishes long-horizon task planning and
can easily generalize to new goals or distinct tasks, even during execution.
However, most traditional methods require predefined module design, which makes
it hard to generalize to different goals. Recent large language model based
approaches can allow for more open-ended planning but often require heavy
prompt engineering or domain-specific pretrained models. To tackle this, we
propose a simple framework that achieves interactive task planning with
language models. Our system incorporates both high-level planning and low-level
function execution via language. We verify the robustness of our system in
generating novel high-level instructions for unseen objectives and its ease of
adaptation to different tasks by merely substituting the task guidelines,
without the need for additional complex prompt engineering. Furthermore, when
the user sends a new request, our system is able to replan accordingly with
precision based on the new request, task guidelines and previously executed
steps. Please check more details on our https://wuphilipp.github.io/itp_site
and https://youtu.be/TrKLuyv26_g.
In-Context Pretraining: Language Modeling Beyond Document Boundaries
Authors: name: Weijia Shi, name: Sewon Min, name: Maria Lomeli, name: Chunting Zhou, name: Margaret Li, name: Victoria Lin, name: Noah A. Smith, name: Luke Zettlemoyer, name: Scott Yih, name: Mike Lewis
Published: 2023-10-16 17:57:12

Large language models (LMs) are currently trained to predict tokens given
document prefixes, enabling them to directly perform long-form generation and
prompting-style tasks which can be reduced to document completion. Existing
pretraining pipelines train LMs by concatenating random sets of short documents
to create input contexts but the prior documents provide no signal for
predicting the next document. We instead present In-Context Pretraining, a new
approach where language models are pretrained on a sequence of related
documents, thereby explicitly encouraging them to read and reason across
document boundaries. We can do In-Context Pretraining by simply changing the
document ordering so that each context contains related documents, and directly
applying existing pretraining pipelines. However, this document sorting problem
is challenging. There are billions of documents and we would like the sort to
maximize contextual similarity for every document without repeating any data.
To do this, we introduce approximate algorithms for finding related documents
with efficient nearest neighbor search and constructing coherent input contexts
with a graph traversal algorithm. Our experiments show In-Context Pretraining
offers a simple and scalable approach to significantly enhance LMs'performance:
we see notable improvements in tasks that require more complex contextual
reasoning, including in-context learning (+8%), reading comprehension (+15%),
faithfulness to previous contexts (+16%), long-context reasoning (+5%), and
retrieval augmentation (+9%).
Llemma: An Open Language Model For Mathematics
Authors: name: Zhangir Azerbayev, name: Hailey Schoelkopf, name: Keiran Paster, name: Marco Dos Santos, name: Stephen McAleer, name: Albert Q. Jiang, name: Jia Deng, name: Stella Biderman, name: Sean Welleck
Published: 2023-10-16 17:54:07

We present Llemma, a large language model for mathematics. We continue
pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web
data containing mathematics, and mathematical code, yielding Llemma. On the
MATH benchmark Llemma outperforms all known open base models, as well as the
unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is
capable of tool use and formal theorem proving without any further finetuning.
We openly release all artifacts, including 7 billion and 34 billion parameter
models, the Proof-Pile-2, and code to replicate our experiments.
Video Language Planning
Authors: name: Yilun Du, name: Mengjiao Yang, name: Pete Florence, name: Fei Xia, name: Ayzaan Wahid, name: Brian Ichter, name: Pierre Sermanet, name: Tianhe Yu, name: Pieter Abbeel, name: Joshua B. Tenenbaum, name: Leslie Kaelbling, name: Andy Zeng, name: Jonathan Tompson
Published: 2023-10-16 17:48:45

We are interested in enabling visual planning for complex long-horizon tasks
in the space of generated videos and language, leveraging recent advances in
large generative models pretrained on Internet-scale data. To this end, we
present video language planning (VLP), an algorithm that consists of a tree
search procedure, where we train (i) vision-language models to serve as both
policies and value functions, and (ii) text-to-video models as dynamics models.
VLP takes as input a long-horizon task instruction and current image
observation, and outputs a long video plan that provides detailed multimodal
(video and language) specifications that describe how to complete the final
task. VLP scales with increasing computation budget where more computation time
results in improved video plans, and is able to synthesize long-horizon video
plans across different robotics domains: from multi-object rearrangement, to
multi-camera bi-arm dexterous manipulation. Generated video plans can be
translated into real robot actions via goal-conditioned policies, conditioned
on each intermediate frame of the generated video. Experiments show that VLP
substantially improves long-horizon task success rates compared to prior
methods on both simulated and real robots (across 3 hardware platforms).
Microscaling Data Formats for Deep Learning
Authors: name: Bita Darvish Rouhani, name: Ritchie Zhao, name: Ankit More, name: Mathew Hall, name: Alireza Khodamoradi, name: Summer Deng, name: Dhruv Choudhary, name: Marius Cornea, name: Eric Dellinger, name: Kristof Denolf, name: Stosic Dusan, name: Venmugil Elango, name: Maximilian Golub, name: Alexander Heinecke, name: Phil James-Roxby, name: Dharmesh Jani, name: Gaurav Kolhe, name: Martin Langhammer, name: Ada Li, name: Levi Melnick, name: Maral Mesmakhosroshahi, name: Andres Rodriguez, name: Michael Schulte, name: Rasoul Shafipour, name: Lei Shao, name: Michael Siu, name: Pradeep Dubey, name: Paulius Micikevicius, name: Maxim Naumov, name: Colin Verilli, name: Ralph Wittig, name: Eric Chung
Published: 2023-10-16 16:07:41

Narrow bit-width data formats are key to reducing the computational and
storage costs of modern deep learning applications. This paper evaluates
Microscaling (MX) data formats that combine a per-block scaling factor with
narrow floating-point and integer types for individual elements.MX formats
balance the competing needs of hardware efficiency, model accuracy, and user
friction. Empirical results on over two dozen benchmarks demonstrate
practicality of MX data formats as a drop-in replacement for baseline FP32 for
AI inference and training with low user friction. We also show the first
instance of training generative language models at sub-8-bit weights,
activations, and gradients with minimal accuracy loss and no modifications to
the training recipe.
Improving Large Language Model Fine-tuning for Solving Math Problems
Authors: name: Yixin Liu, name: Avi Singh, name: C. Daniel Freeman, name: John D. Co-Reyes, name: Peter J. Liu
Published: 2023-10-16 04:11:19

Despite their success in many natural language tasks, solving math problems
remains a significant challenge for large language models (LLMs). A large gap
exists between LLMs' pass-at-one and pass-at-N performance in solving math
problems, suggesting LLMs might be close to finding correct solutions,
motivating our exploration of fine-tuning methods to unlock LLMs' performance.
Using the challenging MATH dataset, we investigate three fine-tuning
strategies: (1) solution fine-tuning, where we fine-tune to generate a detailed
solution for a given math problem; (2) solution-cluster re-ranking, where the
LLM is fine-tuned as a solution verifier/evaluator to choose among generated
candidate solution clusters; (3) multi-task sequential fine-tuning, which
integrates both solution generation and evaluation tasks together efficiently
to enhance the LLM performance. With these methods, we present a thorough
empirical study on a series of PaLM 2 models and find: (1) The quality and
style of the step-by-step solutions used for fine-tuning can make a significant
impact on the model performance; (2) While solution re-ranking and majority
voting are both effective for improving the model performance when used
separately, they can also be used together for an even greater performance
boost; (3) Multi-task fine-tuning that sequentially separates the solution
generation and evaluation tasks can offer improved performance compared with
the solution fine-tuning baseline. Guided by these insights, we design a
fine-tuning recipe that yields approximately 58.8% accuracy on the MATH dataset
with fine-tuned PaLM 2-L models, an 11.2% accuracy improvement over the
few-shot performance of pre-trained PaLM 2-L model with majority voting.
Farzi Data: Autoregressive Data Distillation
Authors: name: Noveen Sachdeva, name: Zexue He, name: Wang-Cheng Kang, name: Jianmo Ni, name: Derek Zhiyuan Cheng, name: Julian McAuley
Published: 2023-10-15 23:23:27

We study data distillation for auto-regressive machine learning tasks, where
the input and output have a strict left-to-right causal structure. More
specifically, we propose Farzi, which summarizes an event sequence dataset into
a small number of synthetic sequences -- Farzi Data -- which are optimized to
maintain (if not improve) model performance compared to training on the full
dataset. Under the hood, Farzi conducts memory-efficient data distillation by
(i) deriving efficient reverse-mode differentiation of the Adam optimizer by
leveraging Hessian-Vector Products; and (ii) factorizing the high-dimensional
discrete event-space into a latent-space which provably promotes implicit
regularization. Empirically, for sequential recommendation and language
modeling tasks, we are able to achieve 98-120% of downstream full-data
performance when training state-of-the-art models on Farzi Data of size as
little as 0.1% of the original dataset. Notably, being able to train better
models with significantly less data sheds light on the design of future large
auto-regressive models, and opens up new opportunities to further scale up
model and data sizes.
When can transformers reason with abstract symbols?
Authors: name: Enric Boix-Adsera, name: Omid Saremi, name: Emmanuel Abbe, name: Samy Bengio, name: Etai Littwin, name: Joshua Susskind
Published: 2023-10-15 06:45:38

We investigate the capabilities of transformer large language models (LLMs)
on relational reasoning tasks involving abstract symbols. Such tasks have long
been studied in the neuroscience literature as fundamental building blocks for
more complex abilities in programming, mathematics, and verbal reasoning. For
(i) regression tasks, we prove that transformers generalize when trained, but
require astonishingly large quantities of training data. For (ii)
next-token-prediction tasks with symbolic labels, we show an "inverse scaling
law": transformers fail to generalize as their embedding dimension increases.
For both settings (i) and (ii), we propose subtle transformer modifications
which can reduce the amount of data needed by adding two trainable parameters
per head.
Reward-Augmented Decoding: Efficient Controlled Text Generation With a
  Unidirectional Reward Model
Authors: name: Haikang Deng, name: Colin Raffel
Published: 2023-10-14 07:19:47

While large language models have proven effective in a huge range of
downstream applications, they often generate text that is problematic or lacks
a desired attribute. In this paper, we introduce Reward-Augmented Decoding
(RAD), a text generation procedure that uses a small unidirectional reward
model to encourage a language model to generate text that has certain
properties. Specifically, RAD uses the reward model to score generations as
they are produced and rescales sampling probabilities to favor high-reward
tokens. By using a unidirectional reward model, RAD can cache activations from
prior generation steps to decrease computational overhead. Through experiments
on generating non-toxic and sentiment-controlled text, we demonstrate that RAD
performs best among methods that change only the generation procedure and
matches the performance of state-of-the-art methods that involve re-training
the language model. We further validate that RAD is effective on very large
language models while incurring a minimal computational overhead.
MiniGPT-v2: large language model as a unified interface for
  vision-language multi-task learning
Authors: name: Jun Chen, name: Deyao Zhu, name: Xiaoqian Shen, name: Xiang Li, name: Zechun Liu, name: Pengchuan Zhang, name: Raghuraman Krishnamoorthi, name: Vikas Chandra, name: Yunyang Xiong, name: Mohamed Elhoseiny
Published: 2023-10-14 03:22:07

Large language models have shown their remarkable capabilities as a general
interface for various language-related applications. Motivated by this, we
target to build a unified interface for completing many vision-language tasks
including image description, visual question answering, and visual grounding,
among others. The challenge is to use a single model for performing diverse
vision-language tasks effectively with simple multi-modal instructions. Towards
this objective, we introduce MiniGPT-v2, a model that can be treated as a
unified interface for better handling various vision-language tasks. We propose
using unique identifiers for different tasks when training the model. These
identifiers enable our model to better distinguish each task instruction
effortlessly and also improve the model learning efficiency for each task.
After the three-stage training, the experimental results show that MiniGPT-v2
achieves strong performance on many visual question-answering and visual
grounding benchmarks compared to other vision-language generalist models. Our
model and codes are available at https://minigpt-v2.github.io/
Ranking LLM-Generated Loop Invariants for Program Verification
Authors: name: Saikat Chakraborty, name: Shuvendu K. Lahiri, name: Sarah Fakhoury, name: Madanlal Musuvathi, name: Akash Lal, name: Aseem Rastogi, name: Aditya Senthilnathan, name: Rahul Sharma, name: Nikhil Swamy
Published: 2023-10-13 18:13:52

Synthesizing inductive loop invariants is fundamental to automating program
verification. In this work, we observe that Large Language Models (such as
gpt-3.5 or gpt-4) are capable of synthesizing loop invariants for a class of
programs in a 0-shot setting, yet require several samples to generate the
correct invariants. This can lead to a large number of calls to a program
verifier to establish an invariant. To address this issue, we propose a {\it
re-ranking} approach for the generated results of LLMs. We have designed a
ranker that can distinguish between correct inductive invariants and incorrect
attempts based on the problem definition. The ranker is optimized as a
contrastive ranker. Experimental results demonstrate that this re-ranking
mechanism significantly improves the ranking of correct invariants among the
generated candidates, leading to a notable reduction in the number of calls to
a verifier.
Can GPT models be Financial Analysts? An Evaluation of ChatGPT and GPT-4
  on mock CFA Exams
Authors: name: Ethan Callanan, name: Amarachi Mbakwe, name: Antony Papadimitriou, name: Yulong Pei, name: Mathieu Sibue, name: Xiaodan Zhu, name: Zhiqiang Ma, name: Xiaomo Liu, name: Sameena Shah
Published: 2023-10-12 19:28:57

Large Language Models (LLMs) have demonstrated remarkable performance on a
wide range of Natural Language Processing (NLP) tasks, often matching or even
beating state-of-the-art task-specific models. This study aims at assessing the
financial reasoning capabilities of LLMs. We leverage mock exam questions of
the Chartered Financial Analyst (CFA) Program to conduct a comprehensive
evaluation of ChatGPT and GPT-4 in financial analysis, considering Zero-Shot
(ZS), Chain-of-Thought (CoT), and Few-Shot (FS) scenarios. We present an
in-depth analysis of the models' performance and limitations, and estimate
whether they would have a chance at passing the CFA exams. Finally, we outline
insights into potential strategies and improvements to enhance the
applicability of LLMs in finance. In this perspective, we hope this work paves
the way for future studies to continue enhancing LLMs for financial reasoning
through rigorous evaluation.
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Authors: name: Xi Chen, name: Xiao Wang, name: Lucas Beyer, name: Alexander Kolesnikov, name: Jialin Wu, name: Paul Voigtlaender, name: Basil Mustafa, name: Sebastian Goodman, name: Ibrahim Alabdulmohsin, name: Piotr Padlewski, name: Daniel Salz, name: Xi Xiong, name: Daniel Vlasic, name: Filip Pavetic, name: Keran Rong, name: Tianli Yu, name: Daniel Keysers, name: Xiaohua Zhai, name: Radu Soricut
Published: 2023-10-13 15:45:19

This paper presents PaLI-3, a smaller, faster, and stronger vision language
model (VLM) that compares favorably to similar models that are 10x larger. As
part of arriving at this strong performance, we compare Vision Transformer
(ViT) models pretrained using classification objectives to contrastively
(SigLIP) pretrained ones. We find that, while slightly underperforming on
standard image classification benchmarks, SigLIP-based PaLI shows superior
performance across various multimodal benchmarks, especially on localization
and visually-situated text understanding. We scale the SigLIP image encoder up
to 2 billion parameters, and achieves a new state-of-the-art on multilingual
cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles
research on fundamental pieces of complex VLMs, and could fuel a new generation
of scaled-up models.
The Consensus Game: Language Model Generation via Equilibrium Search
Authors: name: Athul Paul Jacob, name: Yikang Shen, name: Gabriele Farina, name: Jacob Andreas
Published: 2023-10-13 14:27:21

When applied to question answering and other text generation tasks, language
models (LMs) may be queried generatively (by sampling answers from their output
distribution) or discriminatively (by using them to score or rank a set of
candidate outputs). These procedures sometimes yield very different
predictions. How do we reconcile mutually incompatible scoring procedures to
obtain coherent LM predictions? We introduce a new, a training-free,
game-theoretic procedure for language model decoding. Our approach casts
language model decoding as a regularized imperfect-information sequential
signaling game - which we term the CONSENSUS GAME - in which a GENERATOR seeks
to communicate an abstract correctness parameter using natural language
sentences to a DISCRIMINATOR. We develop computational procedures for finding
approximate equilibria of this game, resulting in a decoding algorithm we call
EQUILIBRIUM-RANKING. Applied to a large number of tasks (including reading
comprehension, commonsense reasoning, mathematical problem-solving, and
dialog), EQUILIBRIUM-RANKING consistently, and sometimes substantially,
improves performance over existing LM decoding procedures - on multiple
benchmarks, we observe that applying EQUILIBRIUM-RANKING to LLaMA-7B
outperforms the much larger LLaMA-65B and PaLM-540B models. These results
highlight the promise of game-theoretic tools for addressing fundamental
challenges of truthfulness and consistency in LMs.
CodeChain: Towards Modular Code Generation Through Chain of
  Self-revisions with Representative Sub-modules
Authors: name: Hung Le, name: Hailin Chen, name: Amrita Saha, name: Akash Gokul, name: Doyen Sahoo, name: Shafiq Joty
Published: 2023-10-13 10:17:48

Large Language Models (LLMs) have already become quite proficient at solving
simpler programming tasks like those in HumanEval or MBPP benchmarks. However,
solving more complex and competitive programming tasks is still quite
challenging for these models - possibly due to their tendency to generate
solutions as monolithic code blocks instead of decomposing them into logical
sub-tasks and sub-modules. On the other hand, experienced programmers
instinctively write modularized code with abstraction for solving complex
tasks, often reusing previously developed modules. To address this gap, we
propose CodeChain, a novel framework for inference that elicits modularized
code generation through a chain of self-revisions, each being guided by some
representative sub-modules generated in previous iterations. Concretely,
CodeChain first instructs the LLM to generate modularized codes through
chain-of-thought prompting. Then it applies a chain of self-revisions by
iterating the two steps: 1) extracting and clustering the generated sub-modules
and selecting the cluster representatives as the more generic and re-usable
implementations, and 2) augmenting the original chain-of-thought prompt with
these selected module-implementations and instructing the LLM to re-generate
new modularized solutions. We find that by naturally encouraging the LLM to
reuse the previously developed and verified sub-modules, CodeChain can
significantly boost both modularity as well as correctness of the generated
solutions, achieving relative pass@1 improvements of 35% on APPS and 76% on
CodeContests. It is shown to be effective on both OpenAI LLMs as well as
open-sourced LLMs like WizardCoder. We also conduct comprehensive ablation
studies with different methods of prompting, number of clusters, model sizes,
program qualities, etc., to provide useful insights that underpin CodeChain's
success.
A Zero-Shot Language Agent for Computer Control with Structured
  Reflection
Authors: name: Tao Li, name: Gang Li, name: Zhiwei Deng, name: Bryan Wang, name: Yang Li
Published: 2023-10-12 21:53:37

Large language models (LLMs) have shown increasing capacity at planning and
executing a high-level goal in a live computer environment (e.g. MiniWoB++). To
perform a task, recent works often require a model to learn from trace examples
of the task via either supervised learning or few/many-shot prompting. Without
these trace examples, it remains a challenge how an agent can autonomously
learn and improve its control on a computer, which limits the ability of an
agent to perform a new task. We approach this problem with a zero-shot agent
that requires no given expert traces. Our agent plans for executable actions on
a partially observed environment, and iteratively progresses a task by
identifying and learning from its mistakes via self-reflection and structured
thought management. On the easy tasks of MiniWoB++, we show that our zero-shot
agent often outperforms recent SoTAs, with more efficient reasoning. For tasks
with more complexity, our reflective agent performs on par with prior best
models, even though previous works had the advantages of accessing expert
traces or additional screen information.