Orionnnn/hub-stats huggingface dataset:
Reformatted Alignment
Authors: name: Run-Ze Fan, name: Xuefeng Li, name: Haoyang Zou, name: Junlong Li, name: Shwai He, name: Ethan Chern, name: Jiewen Hu, name: Pengfei Liu
Published: 2024-02-19 15:21:58

The quality of finetuning data is crucial for aligning large language models
(LLMs) with human values. Current methods to improve data quality are either
labor-intensive or prone to factual errors caused by LLM hallucinations. This
paper explores elevating the quality of existing instruction data to better
align with human values, introducing a simple and effective approach named
ReAlign, which reformats the responses of instruction data into a format that
better aligns with pre-established criteria and the collated evidence. This
approach minimizes human annotation, hallucination, and the difficulty in
scaling, remaining orthogonal to existing alignment techniques. Experimentally,
ReAlign significantly boosts the general alignment ability, math reasoning,
factuality, and readability of the LLMs.
  Encouragingly, without introducing any additional data or advanced training
techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical
reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy.
Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment
ability measured by the Alpaca dataset. This work highlights the need for
further research into the science and mechanistic interpretability of LLMs. We
have made the associated code and data publicly accessible to support future
studies at https://github.com/GAIR-NLP/ReAlign.
Learning to Learn Faster from Human Feedback with Language Model
  Predictive Control
Authors: name: Jacky Liang, name: Fei Xia, name: Wenhao Yu, name: Andy Zeng, name: Montserrat Gonzalez Arenas, name: Maria Attarian, name: Maria Bauza, name: Matthew Bennice, name: Alex Bewley, name: Adil Dostmohamed, name: Chuyuan Kelly Fu, name: Nimrod Gileadi, name: Marissa Giustina, name: Keerthana Gopalakrishnan, name: Leonard Hasenclever, name: Jan Humplik, name: Jasmine Hsu, name: Nikhil Joshi, name: Ben Jyenis, name: Chase Kew, name: Sean Kirmani, name: Tsang-Wei Edward Lee, name: Kuang-Huei Lee, name: Assaf Hurwitz Michaely, name: Joss Moore, name: Ken Oslund, name: Dushyant Rao, name: Allen Ren, name: Baruch Tabanpour, name: Quan Vuong, name: Ayzaan Wahid, name: Ted Xiao, name: Ying Xu, name: Vincent Zhuang, name: Peng Xu, name: Erik Frey, name: Ken Caluwaerts, name: Tingnan Zhang, name: Brian Ichter, name: Jonathan Tompson, name: Leila Takayama, name: Vincent Vanhoucke, name: Izhak Shafran, name: Maja Mataric, name: Dorsa Sadigh, name: Nicolas Heess, name: Kanishka Rao, name: Nik Stewart, name: Jie Tan, name: Carolina Parada
Published: 2024-02-18 04:16:24

Large language models (LLMs) have been shown to exhibit a wide range of
capabilities, such as writing robot code from language commands -- enabling
non-experts to direct robot behaviors, modify them based on feedback, or
compose them to perform new tasks. However, these capabilities (driven by
in-context learning) are limited to short-term interactions, where users'
feedback remains relevant for only as long as it fits within the context size
of the LLM, and can be forgotten over longer interactions. In this work, we
investigate fine-tuning the robot code-writing LLMs, to remember their
in-context interactions and improve their teachability i.e., how efficiently
they adapt to human inputs (measured by average number of corrections before
the user considers the task successful). Our key observation is that when
human-robot interactions are formulated as a partially observable Markov
decision process (in which human language inputs are observations, and robot
code outputs are actions), then training an LLM to complete previous
interactions can be viewed as training a transition dynamics model -- that can
be combined with classic robotics techniques such as model predictive control
(MPC) to discover shorter paths to success. This gives rise to Language Model
Predictive Control (LMPC), a framework that fine-tunes PaLM 2 to improve its
teachability on 78 tasks across 5 robot embodiments -- improving non-expert
teaching success rates of unseen tasks by 26.9% while reducing the average
number of human corrections from 2.4 to 1.9. Experiments show that LMPC also
produces strong meta-learners, improving the success rate of in-context
learning new tasks on unseen robot embodiments and APIs by 31.5%. See videos,
code, and demos at: https://robot-teaching.github.io/.
OneBit: Towards Extremely Low-bit Large Language Models
Authors: name: Yuzhuang Xu, name: Xu Han, name: Zonghan Yang, name: Shuo Wang, name: Qingfu Zhu, name: Zhiyuan Liu, name: Weidong Liu, name: Wanxiang Che
Published: 2024-02-17 14:26:57

Model quantification uses low bit-width values to represent the weight
matrices of models, which is a promising approach to reduce both storage and
computational overheads of deploying highly anticipated LLMs. However, existing
quantization methods suffer severe performance degradation when the bit-width
is extremely reduced, and thus focus on utilizing 4-bit or 8-bit values to
quantize models. This paper boldly quantizes the weight matrices of LLMs to
1-bit, paving the way for the extremely low bit-width deployment of LLMs. For
this target, we introduce a 1-bit quantization-aware training (QAT) framework
named OneBit, including a novel 1-bit parameter representation method to better
quantize LLMs as well as an effective parameter initialization method based on
matrix decomposition to improve the convergence speed of the QAT framework.
Sufficient experimental results indicate that OneBit achieves good performance
(at least 83% of the non-quantized performance) with robust training processes
when only using 1-bit weight matrices.
CoLLaVO: Crayon Large Language and Vision mOdel
Authors: name: Byung-Kwan Lee, name: Beomchan Park, name: Chae Won Kim, name: Yong Man Ro
Published: 2024-02-17 11:03:02

The remarkable success of Large Language Models (LLMs) and instruction tuning
drives the evolution of Vision Language Models (VLMs) towards a versatile
general-purpose model. Yet, it remains unexplored whether current VLMs
genuinely possess quality object-level image understanding capabilities
determined from 'what objects are in the image?' or 'which object corresponds
to a specified bounding box?'. Our findings reveal that the image understanding
capabilities of current VLMs are strongly correlated with their zero-shot
performance on Vision Language (VL) tasks. This suggests that prioritizing
basic image understanding is crucial for VLMs to excel at VL tasks. To enhance
object-level image understanding, we propose Crayon Large Language and Vision
mOdel (CoLLaVO), which incorporates instruction tuning with crayon prompt as a
new visual prompt tuning scheme based on panoptic color maps. Furthermore, we
present a learning strategy of Dual QLoRA to preserve object-level image
understanding without forgetting it during visual instruction tuning, thereby
achieving a significant leap in zero-shot numerous VL benchmarks.
Speculative Streaming: Fast LLM Inference without Auxiliary Models
Authors: name: Nikhil Bhendawade, name: Irina Belousova, name: Qichen Fu, name: Henry Mason, name: Mohammad Rastegari, name: Mahyar Najibi
Published: 2024-02-16 23:36:43

Speculative decoding is a prominent technique to speed up the inference of a
large target language model based on predictions of an auxiliary draft model.
While effective, in application-specific settings, it often involves
fine-tuning both draft and target models to achieve high acceptance rates. As
the number of downstream tasks grows, these draft models add significant
complexity to inference systems. We propose Speculative Streaming, a
single-model speculative decoding method that fuses drafting into the target
model by changing the fine-tuning objective from next token prediction to
future n-gram prediction. Speculative Streaming speeds up decoding by 1.8 -
3.1X in a diverse set of tasks, such as Summarization, Structured Queries, and
Meaning Representation, without sacrificing generation quality. Additionally,
Speculative Streaming is parameter-efficient. It achieves on-par/higher
speed-ups than Medusa-style architectures while using ~10000X fewer extra
parameters, making it well-suited for resource-constrained devices.
GLoRe: When, Where, and How to Improve LLM Reasoning via Global and
  Local Refinements
Authors: name: Alex Havrilla, name: Sharath Raparthy, name: Christoforus Nalmpantis, name: Jane Dwivedi-Yu, name: Maksym Zhuravinskyi, name: Eric Hambro, name: Roberta Railneau
Published: 2024-02-13 20:16:29

State-of-the-art language models can exhibit impressive reasoning refinement
capabilities on math, science or coding tasks. However, recent work
demonstrates that even the best models struggle to identify when and
where to refine without access to external feedback. Outcome-based Reward
Models (ORMs), trained to predict correctness of the final answer
indicating when to refine, offer one convenient solution for deciding when to
refine. Process Based Reward Models (PRMs), trained to predict
correctness of intermediate steps, can then be used to indicate where to
refine. But they are expensive to train, requiring extensive human annotations.
In this paper, we propose Stepwise ORMs (SORMs) which are trained,
only on synthetic data, to approximate the expected future reward of the
optimal policy or V^{star}. More specifically, SORMs are trained to predict
the correctness of the final answer when sampling the current policy many times
(rather than only once as in the case of ORMs). Our experiments show that SORMs
can more accurately detect incorrect reasoning steps compared to ORMs, thus
improving downstream accuracy when doing refinements. We then train
global refinement models, which take only the question and a draft
solution as input and predict a corrected solution, and local
refinement models which also take as input a critique indicating the location
of the first reasoning error. We generate training data for both models
synthetically by reusing data used to train the SORM. We find combining global
and local refinements, using the ORM as a reranker, significantly outperforms
either one individually, as well as a best of three sample baseline. With this
strategy we can improve the accuracy of a LLaMA-2 13B model (already fine-tuned
with RL) on GSM8K from 53\% to 65\% when greedily sampled.
Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning
Authors: name: Zhiyang Xu, name: Chao Feng, name: Rulin Shao, name: Trevor Ashby, name: Ying Shen, name: Di Jin, name: Yu Cheng, name: Qifan Wang, name: Lifu Huang
Published: 2024-02-18 19:38:44

Despite vision-language models' (VLMs) remarkable capabilities as versatile
visual assistants, two substantial challenges persist within the existing VLM
frameworks: (1) lacking task diversity in pretraining and visual instruction
tuning, and (2) annotation error and bias in GPT-4 synthesized instruction
tuning data. Both challenges lead to issues such as poor generalizability,
hallucination, and catastrophic forgetting. To address these challenges, we
construct Vision-Flan, the most diverse publicly available visual instruction
tuning dataset to date, comprising 187 diverse tasks and 1,664,261 instances
sourced from academic datasets, and each task is accompanied by an
expert-written instruction. In addition, we propose a two-stage instruction
tuning framework, in which VLMs are firstly finetuned on Vision-Flan and
further tuned on GPT-4 synthesized data. We find this two-stage tuning
framework significantly outperforms the traditional single-stage visual
instruction tuning framework and achieves the state-of-the-art performance
across a wide range of multi-modal evaluation benchmarks. Finally, we conduct
in-depth analyses to understand visual instruction tuning and our findings
reveal that: (1) GPT-4 synthesized data does not substantially enhance VLMs'
capabilities but rather modulates the model's responses to human-preferred
formats; (2) A minimal quantity (e.g., 1,000) of GPT-4 synthesized data can
effectively align VLM responses with human-preference; (3) Visual instruction
tuning mainly helps large-language models (LLMs) to understand visual features.
LongAgent: Scaling Language Models to 128k Context through Multi-Agent
  Collaboration
Authors: name: Jun Zhao, name: Can Zu, name: Hao Xu, name: Yi Lu, name: Wei He, name: Yiwen Ding, name: Tao Gui, name: Qi Zhang, name: Xuanjing Huang
Published: 2024-02-18 11:46:52

Large language models (LLMs) have demonstrated impressive performance in
understanding language and executing complex reasoning tasks. However, LLMs
with long context windows have been notorious for their expensive training
costs and high inference latency. Even the most advanced models such as GPT-4
and Claude2 often make mistakes when processing inputs of over 100k tokens, a
phenomenon also known as lost in the middle. In this paper, we propose
LongAgent, a method based on multi-agent collaboration, which scales
LLMs (e.g., LLaMA) to a context of 128K and demonstrates potential superiority
in long-text processing compared to GPT-4. In LongAgent, a leader is
responsible for understanding user intent and directing team members to acquire
information from documents. Due to members' hallucinations, it is non-trivial
for a leader to obtain accurate information from the responses of dozens to
hundreds of members. To address this, we develop an inter-member
communication mechanism to resolve response conflicts caused by hallucinations
through information sharing. Our experimental results indicate that
LongAgent offers a promising alternative for long-text processing. The
agent team instantiated with LLaMA-7B achieves significant improvements in
tasks such as 128k-long text retrieval, multi-hop question answering, compared
to GPT-4.
FinTral: A Family of GPT-4 Level Multimodal Financial Large Language
  Models
Authors: name: Gagan Bhatia, name: El Moatez Billah Nagoudi, name: Hasan Cavusoglu, name: Muhammad Abdul-Mageed
Published: 2024-02-16 05:05:12

We introduce FinTral, a suite of state-of-the-art multimodal large language
models (LLMs) built upon the Mistral-7b model and tailored for financial
analysis. FinTral integrates textual, numerical, tabular, and image data. We
enhance FinTral with domain-specific pretraining, instruction fine-tuning, and
RLAIF training by exploiting a large collection of textual and visual datasets
we curate for this work. We also introduce an extensive benchmark featuring
nine tasks and 25 datasets for evaluation, including hallucinations in the
financial domain. Our FinTral model trained with direct preference optimization
employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R,
demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5
in all tasks and surpasses GPT-4 in five out of nine tasks, marking a
significant advancement in AI-driven financial technology. We also demonstrate
that FinTral has the potential to excel in real-time analysis and
decision-making in diverse financial contexts.
Pushing Auto-regressive Models for 3D Shape Generation at Capacity and
  Scalability
Authors: name: Xuelin Qian, name: Yu Wang, name: Simian Luo, name: Yinda Zhang, name: Ying Tai, name: Zhenyu Zhang, name: Chengjie Wang, name: Xiangyang Xue, name: Bo Zhao, name: Tiejun Huang, name: Yunsheng Wu, name: Yanwei Fu
Published: 2024-02-19 15:33:09

Auto-regressive models have achieved impressive results in 2D image
generation by modeling joint distributions in grid space. In this paper, we
extend auto-regressive models to 3D domains, and seek a stronger ability of 3D
shape generation by improving auto-regressive models at capacity and
scalability simultaneously. Firstly, we leverage an ensemble of publicly
available 3D datasets to facilitate the training of large-scale models. It
consists of a comprehensive collection of approximately 900,000 objects, with
multiple properties of meshes, points, voxels, rendered images, and text
captions. This diverse labeled dataset, termed Objaverse-Mix, empowers our
model to learn from a wide range of object variations. However, directly
applying 3D auto-regression encounters critical challenges of high
computational demands on volumetric grids and ambiguous auto-regressive order
along grid dimensions, resulting in inferior quality of 3D shapes. To this end,
we then present a novel framework Argus3D in terms of capacity. Concretely, our
approach introduces discrete representation learning based on a latent vector
instead of volumetric grids, which not only reduces computational costs but
also preserves essential geometric details by learning the joint distributions
in a more tractable order. The capacity of conditional generation can thus be
realized by simply concatenating various conditioning inputs to the latent
vector, such as point clouds, categories, images, and texts. In addition,
thanks to the simplicity of our model architecture, we naturally scale up our
approach to a larger model with an impressive 3.6 billion parameters, further
enhancing the quality of versatile 3D generation. Extensive experiments on four
generation tasks demonstrate that Argus3D can synthesize diverse and faithful
shapes across multiple categories, achieving remarkable performance.
DiLightNet: Fine-grained Lighting Control for Diffusion-based Image
  Generation
Authors: name: Chong Zeng, name: Yue Dong, name: Pieter Peers, name: Youkang Kong, name: Hongzhi Wu, name: Xin Tong
Published: 2024-02-19 08:17:21

This paper presents a novel method for exerting fine-grained lighting control
during text-driven diffusion-based image generation. While existing diffusion
models already have the ability to generate images under any lighting
condition, without additional guidance these models tend to correlate image
content and lighting. Moreover, text prompts lack the necessary expressional
power to describe detailed lighting setups. To provide the content creator with
fine-grained control over the lighting during image generation, we augment the
text-prompt with detailed lighting information in the form of radiance hints,
i.e., visualizations of the scene geometry with a homogeneous canonical
material under the target lighting. However, the scene geometry needed to
produce the radiance hints is unknown. Our key observation is that we only need
to guide the diffusion process, hence exact radiance hints are not necessary;
we only need to point the diffusion model in the right direction. Based on this
observation, we introduce a three stage method for controlling the lighting
during image generation. In the first stage, we leverage a standard pretrained
diffusion model to generate a provisional image under uncontrolled lighting.
Next, in the second stage, we resynthesize and refine the foreground object in
the generated image by passing the target lighting to a refined diffusion
model, named DiLightNet, using radiance hints computed on a coarse shape of the
foreground object inferred from the provisional image. To retain the texture
details, we multiply the radiance hints with a neural encoding of the
provisional synthesized image before passing it to DiLightNet. Finally, in the
third stage, we resynthesize the background to be consistent with the lighting
on the foreground object. We demonstrate and validate our lighting controlled
diffusion model on a variety of text prompts and lighting conditions.
Universal Manipulation Interface: In-The-Wild Robot Teaching Without
  In-The-Wild Robots
Authors: name: Cheng Chi, name: Zhenjia Xu, name: Chuer Pan, name: Eric Cousineau, name: Benjamin Burchfiel, name: Siyuan Feng, name: Russ Tedrake, name: Shuran Song
Published: 2024-02-15 21:11:50

We present Universal Manipulation Interface (UMI) -- a data collection and
policy learning framework that allows direct skill transfer from in-the-wild
human demonstrations to deployable robot policies. UMI employs hand-held
grippers coupled with careful interface design to enable portable, low-cost,
and information-rich data collection for challenging bimanual and dynamic
manipulation demonstrations. To facilitate deployable policy learning, UMI
incorporates a carefully designed policy interface with inference-time latency
matching and a relative-trajectory action representation. The resulting learned
policies are hardware-agnostic and deployable across multiple robot platforms.
Equipped with these features, UMI framework unlocks new robot manipulation
capabilities, allowing zero-shot generalizable dynamic, bimanual, precise, and
long-horizon behaviors, by only changing the training data for each task. We
demonstrate UMI's versatility and efficacy with comprehensive real-world
experiments, where policies learned via UMI zero-shot generalize to novel
environments and objects when trained on diverse human demonstrations. UMI's
hardware and software system is open-sourced at https://umi-gripper.github.io.
GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object
  with Gaussian Splatting
Authors: name: Chen Yang, name: Sikuang Li, name: Jiemin Fang, name: Ruofan Liang, name: Lingxi Xie, name: Xiaopeng Zhang, name: Wei Shen, name: Qi Tian
Published: 2024-02-15 18:42:33

Reconstructing and rendering 3D objects from highly sparse views is of
critical importance for promoting applications of 3D vision techniques and
improving user experience. However, images from sparse views only contain very
limited 3D information, leading to two significant challenges: 1) Difficulty in
building multi-view consistency as images for matching are too few; 2)
Partially omitted or highly compressed object information as view coverage is
insufficient. To tackle these challenges, we propose GaussianObject, a
framework to represent and render the 3D object with Gaussian splatting, that
achieves high rendering quality with only 4 input images. We first introduce
techniques of visual hull and floater elimination which explicitly inject
structure priors into the initial optimization process for helping build
multi-view consistency, yielding a coarse 3D Gaussian representation. Then we
construct a Gaussian repair model based on diffusion models to supplement the
omitted object information, where Gaussians are further refined. We design a
self-generating strategy to obtain image pairs for training the repair model.
Our GaussianObject is evaluated on several challenging datasets, including
MipNeRF360, OmniObject3D, and OpenIllumination, achieving strong reconstruction
results from only 4 views and significantly outperforming previous
state-of-the-art methods.
PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong
  Vision-language Adapter
Authors: name: Junfei Xiao, name: Zheng Xu, name: Alan Yuille, name: Shen Yan, name: Boyu Wang
Published: 2024-02-16 18:54:47

This paper demonstrates that a progressively aligned language model can
effectively bridge frozen vision encoders and large language models (LLMs).
While the fundamental architecture and pre-training methods of vision encoders
and LLMs have been extensively studied, the architecture and training strategy
of vision-language adapters vary significantly across recent works. Our
research undertakes a thorough exploration of the state-of-the-art perceiver
resampler architecture and builds a strong baseline. However, we observe that
the vision-language alignment with perceiver resampler exhibits slow
convergence and limited scalability with a lack of direct supervision. To
address this issue, we propose PaLM2-VAdapter, employing a progressively
aligned language model as the vision-language adapter. Compared to the strong
baseline with perceiver resampler, our method empirically shows faster
convergence, higher performance, and stronger scalability. Extensive
experiments across various Visual Question Answering (VQA) and captioning tasks
on both images and videos demonstrate that our model exhibits state-of-the-art
visual understanding and multi-modal reasoning capabilities. Notably, our
method achieves these advancements with 30~70% fewer parameters than the
state-of-the-art large vision-language models, marking a significant efficiency
improvement.