Orionnnn/hub-stats huggingface dataset:
Language Models can be Logical Solvers
Authors: name: Jiazhan Feng, name: Ruochen Xu, name: Junheng Hao, name: Hiteshi Sharma, name: Yelong Shen, name: Dongyan Zhao, name: Weizhu Chen
Published: 2023-11-10 16:23:50

Logical reasoning is a fundamental aspect of human intelligence and a key
component of tasks like problem-solving and decision-making. Recent
advancements have enabled Large Language Models (LLMs) to potentially exhibit
reasoning capabilities, but complex logical reasoning remains a challenge. The
state-of-the-art, solver-augmented language models, use LLMs to parse natural
language logical questions into symbolic representations first and then adopt
external logical solvers to take in the symbolic representations and output the
answers. Despite their impressive performance, any parsing errors will
inevitably result in the failure of the execution of the external logical
solver and no answer to the logical questions. In this paper, we introduce
LoGiPT, a novel language model that directly emulates the reasoning processes
of logical solvers and bypasses the parsing errors by learning to strict
adherence to solver syntax and grammar. LoGiPT is fine-tuned on a newly
constructed instruction-tuning dataset derived from revealing and refining the
invisible reasoning process of deductive solvers. Experimental results on two
public deductive reasoning datasets demonstrate that LoGiPT outperforms
state-of-the-art solver-augmented LMs and few-shot prompting methods on
competitive LLMs like ChatGPT or GPT-4.
Lumos: Learning Agents with Unified Data, Modular Design, and
  Open-Source LLMs
Authors: name: Da Yin, name: Faeze Brahman, name: Abhilasha Ravichander, name: Khyathi Chandu, name: Kai-Wei Chang, name: Yejin Choi, name: Bill Yuchen Lin
Published: 2023-11-09 00:30:13

We introduce Lumos, a novel framework for training language agents that
employs a unified data format and a modular architecture based on open-source
large language models (LLMs). Lumos consists of three distinct modules:
planning, grounding, and execution. The planning module breaks down a task into
a series of high-level, tool-agnostic subgoals, which are then made specific by
the grounding module through a set of low-level actions. These actions are
subsequently executed by the execution module, utilizing a range of
off-the-shelf tools and APIs. In order to train these modules effectively,
high-quality annotations of subgoals and actions were collected and are made
available for fine-tuning open-source LLMs for various tasks such as complex
question answering, web tasks, and math problems. Leveraging this unified data
and modular design, Lumos not only achieves comparable or superior performance
to current, state-of-the-art agents, but also exhibits several key advantages:
(1) Lumos surpasses GPT-4/3.5-based agents in complex question answering and
web tasks, while equalling the performance of significantly larger LLM agents
on math tasks; (2) Lumos outperforms open-source agents created through
conventional training methods and those using chain-of-thoughts training; and
(3) Lumos is capable of effectively generalizing to unseen interactive tasks,
outperforming larger LLM-based agents and even exceeding performance of
specialized agents.
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
Authors: name: In Gim, name: Guojun Chen, name: Seung-seob Lee, name: Nikhil Sarda, name: Anurag Khandelwal, name: Lin Zhong
Published: 2023-11-07 18:17:05

We present Prompt Cache, an approach for accelerating inference for large
language models (LLM) by reusing attention states across different LLM prompts.
Many input prompts have overlapping text segments, such as system messages,
prompt templates, and documents provided for context. Our key insight is that
by precomputing and storing the attention states of these frequently occurring
text segments on the inference server, we can efficiently reuse them when these
segments appear in user prompts. Prompt Cache employs a schema to explicitly
define such reusable text segments, called prompt modules. The schema ensures
positional accuracy during attention state reuse and provides users with an
interface to access cached states in their prompt. Using a prototype
implementation, we evaluate Prompt Cache across several LLMs. We show that
Prompt Cache significantly reduce latency in time-to-first-token, especially
for longer prompts such as document-based question answering and
recommendations. The improvements range from 8x for GPU-based inference to 60x
for CPU-based inference, all while maintaining output accuracy and without the
need for model parameter modifications.
On the Road with GPT-4V(ision): Early Explorations of Visual-Language
  Model on Autonomous Driving
Authors: name: Licheng Wen, name: Xuemeng Yang, name: Daocheng Fu, name: Xiaofeng Wang, name: Pinlong Cai, name: Xin Li, name: Tao Ma, name: Yingxuan Li, name: Linran Xu, name: Dengke Shang, name: Zheng Zhu, name: Shaoyan Sun, name: Yeqi Bai, name: Xinyu Cai, name: Min Dou, name: Shuanglu Hu, name: Botian Shi
Published: 2023-11-09 12:58:37

The pursuit of autonomous driving technology hinges on the sophisticated
integration of perception, decision-making, and control systems. Traditional
approaches, both data-driven and rule-based, have been hindered by their
inability to grasp the nuance of complex driving environments and the
intentions of other road users. This has been a significant bottleneck,
particularly in the development of common sense reasoning and nuanced scene
understanding necessary for safe and reliable autonomous driving. The advent of
Visual Language Models (VLM) represents a novel frontier in realizing fully
autonomous vehicle driving. This report provides an exhaustive evaluation of
the latest state-of-the-art VLM, \modelnamefull, and its application in
autonomous driving scenarios. We explore the model's abilities to understand
and reason about driving scenes, make decisions, and ultimately act in the
capacity of a driver. Our comprehensive tests span from basic scene recognition
to complex causal reasoning and real-time decision-making under varying
conditions. Our findings reveal that \modelname demonstrates superior
performance in scene understanding and causal reasoning compared to existing
autonomous systems. It showcases the potential to handle out-of-distribution
scenarios, recognize intentions, and make informed decisions in real driving
contexts. However, challenges remain, particularly in direction discernment,
traffic light recognition, vision grounding, and spatial reasoning tasks. These
limitations underscore the need for further research and development. Project
is now available on GitHub for interested parties to access and utilize:
https://github.com/PJLab-ADG/GPT4V-AD-Exploration
LCM-LoRA: A Universal Stable-Diffusion Acceleration Module
Authors: name: Simian Luo, name: Yiqin Tan, name: Suraj Patil, name: Daniel Gu, name: Patrick von Platen, name: Apolinário Passos, name: Longbo Huang, name: Jian Li, name: Hang Zhao
Published: 2023-11-09 18:04:15

Latent Consistency Models (LCMs) have achieved impressive performance in
accelerating text-to-image generative tasks, producing high-quality images with
minimal inference steps. LCMs are distilled from pre-trained latent diffusion
models (LDMs), requiring only ~32 A100 GPU training hours. This report further
extends LCMs' potential in two aspects: First, by applying LoRA distillation to
Stable-Diffusion models including SD-V1.5, SSD-1B, and SDXL, we have expanded
LCM's scope to larger models with significantly less memory consumption,
achieving superior image generation quality. Second, we identify the LoRA
parameters obtained through LCM distillation as a universal Stable-Diffusion
acceleration module, named LCM-LoRA. LCM-LoRA can be directly plugged into
various Stable-Diffusion fine-tuned models or LoRAs without training, thus
representing a universally applicable accelerator for diverse image generation
tasks. Compared with previous numerical PF-ODE solvers such as DDIM,
DPM-Solver, LCM-LoRA can be viewed as a plug-in neural PF-ODE solver that
possesses strong generalization abilities. Project page:
https://github.com/luosiallen/latent-consistency-model.
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Authors: name: Shilong Liu, name: Hao Cheng, name: Haotian Liu, name: Hao Zhang, name: Feng Li, name: Tianhe Ren, name: Xueyan Zou, name: Jianwei Yang, name: Hang Su, name: Jun Zhu, name: Lei Zhang, name: Jianfeng Gao, name: Chunyuan Li
Published: 2023-11-09 15:22:26

LLaVA-Plus is a general-purpose multimodal assistant that expands the
capabilities of large multimodal models. It maintains a skill repository of
pre-trained vision and vision-language models and can activate relevant tools
based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on
multimodal instruction-following data to acquire the ability to use tools,
covering visual understanding, generation, external knowledge retrieval, and
compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in
existing capabilities and exhibits new ones. It is distinct in that the image
query is directly grounded and actively engaged throughout the entire human-AI
interaction sessions, significantly improving tool use performance and enabling
new scenarios.
u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model
Authors: name: Jinjin Xu, name: Liwu Xu, name: Yuzhe Yang, name: Xiang Li, name: Yanchun Xie, name: Yi-Jie Huang, name: Yaqian Li
Published: 2023-11-09 13:18:27

Recent advances such as LLaVA and Mini-GPT4 have successfully integrated
visual information into LLMs, yielding inspiring outcomes and giving rise to a
new generation of multi-modal LLMs, or MLLMs. Nevertheless, these methods
struggle with hallucinations and the mutual interference between tasks. To
tackle these problems, we propose an efficient and accurate approach to adapt
to downstream tasks by utilizing LLM as a bridge to connect multiple expert
models, namely u-LLaVA. Firstly, we incorporate the modality alignment module
and multi-task modules into LLM. Then, we reorganize or rebuild multi-type
public datasets to enable efficient modality alignment and instruction
following. Finally, task-specific information is extracted from the trained LLM
and provided to different modules for solving downstream tasks. The overall
framework is simple, effective, and achieves state-of-the-art performance
across multiple benchmarks. We also release our model, the generated data, and
the code base publicly available.
GPT4All: An Ecosystem of Open Source Compressed Language Models
Authors: name: Yuvanesh Anand, name: Zach Nussbaum, name: Adam Treat, name: Aaron Miller, name: Richard Guo, name: Ben Schmidt, name: GPT4All Community, name: Brandon Duderstadt, name: Andriy Mulyar
Published: 2023-11-06 23:50:20

Large language models (LLMs) have recently achieved human-level performance
on a range of professional and academic benchmarks. The accessibility of these
models has lagged behind their performance. State-of-the-art LLMs require
costly infrastructure; are only accessible via rate-limited, geo-locked, and
censored web interfaces; and lack publicly available code and technical
reports. In this paper, we tell the story of GPT4All, a popular open source
repository that aims to democratize access to LLMs. We outline the technical
details of the original GPT4All model family, as well as the evolution of the
GPT4All project from a single model into a fully fledged open source ecosystem.
It is our hope that this paper acts as both a technical overview of the
original GPT4All models as well as a case study on the subsequent growth of the
GPT4All open source ecosystem.
GENOME: GenerativE Neuro-symbOlic visual reasoning by growing and
  reusing ModulEs
Authors: name: Zhenfang Chen, name: Rui Sun, name: Wenjun Liu, name: Yining Hong, name: Chuang Gan
Published: 2023-11-08 18:59:05

Recent works have shown that Large Language Models (LLMs) could empower
traditional neuro-symbolic models via programming capabilities to translate
language into module descriptions, thus achieving strong visual reasoning
results while maintaining the model's transparency and efficiency. However,
these models usually exhaustively generate the entire code snippet given each
new instance of a task, which is extremely ineffective. We propose generative
neuro-symbolic visual reasoning by growing and reusing modules. Specifically,
our model consists of three unique stages, module initialization, module
generation, and module execution. First, given a vision-language task, we adopt
LLMs to examine whether we could reuse and grow over established modules to
handle this new task. If not, we initialize a new module needed by the task and
specify the inputs and outputs of this new module. After that, the new module
is created by querying LLMs to generate corresponding code snippets that match
the requirements. In order to get a better sense of the new module's ability,
we treat few-shot training examples as test cases to see if our new module
could pass these cases. If yes, the new module is added to the module library
for future reuse. Finally, we evaluate the performance of our model on the
testing set by executing the parsed programs with the newly made visual modules
to get the results. We find the proposed model possesses several advantages.
First, it performs competitively on standard tasks like visual question
answering and referring expression comprehension; Second, the modules learned
from one task can be seamlessly transferred to new tasks; Last but not least,
it is able to adapt to new visual reasoning tasks by observing a few training
examples and reusing modules.
3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
Authors: name: Chenfeng Xu, name: Huan Ling, name: Sanja Fidler, name: Or Litany
Published: 2023-11-07 23:46:41

We present 3DiffTection, a state-of-the-art method for 3D object detection
from single images, leveraging features from a 3D-aware diffusion model.
Annotating large-scale image data for 3D detection is resource-intensive and
time-consuming. Recently, pretrained large image diffusion models have become
prominent as effective feature extractors for 2D perception tasks. However,
these features are initially trained on paired text and image data, which are
not optimized for 3D tasks, and often exhibit a domain gap when applied to the
target data. Our approach bridges these gaps through two specialized tuning
strategies: geometric and semantic. For geometric tuning, we fine-tune a
diffusion model to perform novel view synthesis conditioned on a single image,
by introducing a novel epipolar warp operator. This task meets two essential
criteria: the necessity for 3D awareness and reliance solely on posed image
data, which are readily available (e.g., from videos) and does not require
manual annotation. For semantic refinement, we further train the model on
target data with detection supervision. Both tuning phases employ ControlNet to
preserve the integrity of the original feature capabilities. In the final step,
we harness these enhanced capabilities to conduct a test-time prediction
ensemble across multiple virtual viewpoints. Through our methodology, we obtain
3D-aware features that are tailored for 3D detection and excel in identifying
cross-view point correspondences. Consequently, our model emerges as a powerful
3D detector, substantially surpassing previous benchmarks, e.g., Cube-RCNN, a
precedent in single-view 3D detection by 9.43\% in AP3D on the
Omni3D-ARkitscene dataset. Furthermore, 3DiffTection showcases robust data
efficiency and generalization to cross-domain data.
LRM: Large Reconstruction Model for Single Image to 3D
Authors: name: Yicong Hong, name: Kai Zhang, name: Jiuxiang Gu, name: Sai Bi, name: Yang Zhou, name: Difan Liu, name: Feng Liu, name: Kalyan Sunkavalli, name: Trung Bui, name: Hao Tan
Published: 2023-11-08 00:03:52

We propose the first Large Reconstruction Model (LRM) that predicts the 3D
model of an object from a single input image within just 5 seconds. In contrast
to many previous methods that are trained on small-scale datasets such as
ShapeNet in a category-specific fashion, LRM adopts a highly scalable
transformer-based architecture with 500 million learnable parameters to
directly predict a neural radiance field (NeRF) from the input image. We train
our model in an end-to-end manner on massive multi-view data containing around
1 million objects, including both synthetic renderings from Objaverse and real
captures from MVImgNet. This combination of a high-capacity model and
large-scale training data empowers our model to be highly generalizable and
produce high-quality 3D reconstructions from various testing inputs including
real-world in-the-wild captures and images from generative models. Video demos
and interactable 3D meshes can be found on this website:
https://yiconghong.me/LRM/.
Holistic Evaluation of Text-To-Image Models
Authors: name: Tony Lee, name: Michihiro Yasunaga, name: Chenlin Meng, name: Yifan Mai, name: Joon Sung Park, name: Agrim Gupta, name: Yunzhi Zhang, name: Deepak Narayanan, name: Hannah Benita Teufel, name: Marco Bellagente, name: Minguk Kang, name: Taesung Park, name: Jure Leskovec, name: Jun-Yan Zhu, name: Li Fei-Fei, name: Jiajun Wu, name: Stefano Ermon, name: Percy Liang
Published: 2023-11-07 19:00:56

The stunning qualitative improvement of recent text-to-image models has led
to their widespread attention and adoption. However, we lack a comprehensive
quantitative understanding of their capabilities and risks. To fill this gap,
we introduce a new benchmark, Holistic Evaluation of Text-to-Image Models
(HEIM). Whereas previous evaluations focus mostly on text-image alignment and
image quality, we identify 12 aspects, including text-image alignment, image
quality, aesthetics, originality, reasoning, knowledge, bias, toxicity,
fairness, robustness, multilinguality, and efficiency. We curate 62 scenarios
encompassing these aspects and evaluate 26 state-of-the-art text-to-image
models on this benchmark. Our results reveal that no single model excels in all
aspects, with different models demonstrating different strengths. We release
the generated images and human evaluation results for full transparency at
https://crfm.stanford.edu/heim/v1.1.0 and the code at
https://github.com/stanford-crfm/helm, which is integrated with the HELM
codebase.
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
Authors: name: Zhen Yang, name: Yingxue Zhang, name: Fandong Meng, name: Jie Zhou
Published: 2023-11-08 10:34:16

Despite Multi-modal Large Language Models (MM-LLMs) have made exciting
strides recently, they are still struggling to efficiently model the
interactions among multi-modal inputs and the generation in non-textual
modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an
approach to treat the input from any modality as a token sequence and learn a
joint embedding space for all modalities. Specifically, for the input from any
modality, TEAL first discretizes it into a token sequence with the
off-the-shelf tokenizer and embeds the token sequence into a joint embedding
space with a learnable embedding matrix. MM-LLMs just need to predict the
multi-modal tokens autoregressively as the textual LLMs do. Finally, the
corresponding de-tokenizer is applied to generate the output in each modality
based on the predicted token sequence. With the joint embedding space, TEAL
enables the frozen LLMs to perform both understanding and generation tasks
involving non-textual modalities, such as image and audio. Thus, the textual
LLM can just work as an interface and maintain its high performance in textual
understanding and generation. Experiments show that TEAL achieves substantial
improvements in multi-modal understanding, and implements a simple scheme for
multi-modal generations.
NExT-Chat: An LMM for Chat, Detection and Segmentation
Authors: name: Ao Zhang, name: Liming Zhao, name: Chen-Wei Xie, name: Yun Zheng, name: Wei Ji, name: Tat-Seng Chua
Published: 2023-11-08 07:15:05

The development of large language models (LLMs) has greatly advanced the
field of multimodal understanding, leading to the emergence of large multimodal
models (LMMs). In order to enhance the level of visual comprehension, recent
studies have equipped LMMs with region-level understanding capabilities by
representing object bounding box coordinates as a series of text sequences
(pixel2seq). In this paper, we introduce a novel paradigm for object location
modeling called pixel2emb method, where we ask the LMM to output the location
embeddings and then decoded by different decoders. This paradigm allows for
different location formats (such as bounding boxes and masks) to be used in
multimodal conversations Furthermore, this kind of embedding based location
modeling enables the utilization of existing practices in localization tasks,
such as detection and segmentation. In scenarios with limited resources, our
pixel2emb demonstrates superior performance compared to existing
state-of-the-art (SOTA) approaches in both the location input and output tasks
under fair comparison. Leveraging the proposed pixel2emb method, we train an
LMM named NExT-Chat and demonstrate its capability of handling multiple tasks
like visual grounding, region caption, and grounded reasoning.
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with
  Modality Collaboration
Authors: name: Qinghao Ye, name: Haiyang Xu, name: Jiabo Ye, name: Ming Yan, name: Haowei Liu, name: Qi Qian, name: Ji Zhang, name: Fei Huang, name: Jingren Zhou, name: Anwen HU
Published: 2023-11-07 14:21:29

Multi-modal Large Language Models (MLLMs) have demonstrated impressive
instruction abilities across various open-ended tasks. However, previous
methods primarily focus on enhancing multi-modal capabilities. In this work, we
introduce a versatile multi-modal large language model, mPLUG-Owl2, which
effectively leverages modality collaboration to improve performance in both
text and multi-modal tasks. mPLUG-Owl2 utilizes a modularized network design,
with the language decoder acting as a universal interface for managing
different modalities. Specifically, mPLUG-Owl2 incorporates shared functional
modules to facilitate modality collaboration and introduces a modality-adaptive
module that preserves modality-specific features. Extensive experiments reveal
that mPLUG-Owl2 is capable of generalizing both text tasks and multi-modal
tasks and achieving state-of-the-art performances with a single generic model.
Notably, mPLUG-Owl2 is the first MLLM model that demonstrates the modality
collaboration phenomenon in both pure-text and multi-modal scenarios, setting a
pioneering path in the development of future multi-modal foundation models.
Everything of Thoughts: Defying the Law of Penrose Triangle for Thought
  Generation
Authors: name: Ruomeng Ding, name: Chaoyun Zhang, name: Lu Wang, name: Yong Xu, name: Minghua Ma, name: Wei Zhang, name: Si Qin, name: Saravan Rajmohan, name: Qingwei Lin, name: Dongmei Zhang
Published: 2023-11-07 12:30:36

Recent advancements in Large Language Models (LLMs) have revolutionized
decision-making by breaking down complex problems into more manageable language
sequences referred to as ``thoughts''. An effective thought design should
consider three key perspectives: performance, efficiency, and flexibility.
However, existing thought can at most exhibit two of these attributes. To
address these limitations, we introduce a novel thought prompting approach
called ``Everything of Thoughts'' (XoT) to defy the law of ``Penrose triangle
of existing thought paradigms. XoT leverages pretrained reinforcement learning
and Monte Carlo Tree Search (MCTS) to incorporate external domain knowledge
into thoughts, thereby enhancing LLMs' capabilities and enabling them to
generalize to unseen problems efficiently. Through the utilization of the
MCTS-LLM collaborative thought revision framework, this approach autonomously
produces high-quality comprehensive cognitive mappings with minimal LLM
interactions. Additionally, XoT empowers LLMs to engage in unconstrained
thinking, allowing for flexible cognitive mappings for problems with multiple
solutions.