Orionnnn/hub-stats huggingface dataset:
Principled Instructions Are All You Need for Questioning LLaMA-1/2,
  GPT-3.5/4
Authors: name: Sondos Mahmoud Bsharat, name: Aidar Myrzakhan, name: Zhiqiang Shen
Published: 2023-12-26 18:59:33

This paper introduces 26 guiding principles designed to streamline the
process of querying and prompting large language models. Our goal is to
simplify the underlying concepts of formulating questions for various scales of
large language models, examining their abilities, and enhancing user
comprehension on the behaviors of different scales of large language models
when feeding into different prompts. Extensive experiments are conducted on
LLaMA-1/2 (7B, 13B and 70B), GPT-3.5/4 to verify the effectiveness of the
proposed principles on instructions and prompts design. We hope that this work
can provide a better guide for researchers working on the prompting of large
language models. Project page is available at
https://github.com/VILA-Lab/ATLAS.
SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective
  Depth Up-Scaling
Authors: name: Dahyun Kim, name: Chanjun Park, name: Sanghoon Kim, name: Wonsung Lee, name: Wonho Song, name: Yunsu Kim, name: Hyeonwoo Kim, name: Yungi Kim, name: Hyeonju Lee, name: Jihoo Kim, name: Changbae Ahn, name: Seonghoon Yang, name: Sukyung Lee, name: Hyunbyung Park, name: Gyoungjin Gim, name: Mikyoung Cha, name: Hwalsuk Lee, name: Sunghun Kim
Published: 2023-12-23 05:11:37

We introduce depth up-scaling (DUS), a novel technique to up-scale base LLMs
efficiently and effectively in a simple manner. In contrast to
mixture-of-experts (MoE), DUS does not require complex changes to train and
inference. Using DUS, we build SOLAR 10.7B, a large language model (LLM) with
10.7 billion parameters, demonstrating superior performance in various natural
language processing (NLP) tasks. Comparative evaluations show that SOLAR 10.7B
outperforms existing open-source pretrained LLMs, such as Llama 2 and Mistral
7B. We additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for
instruction-following capabilities, surpassing Mixtral-8x7B. SOLAR 10.7B is
publicly available under the Apache 2.0 license, promoting broad access and
application in the LLM field.
Supervised Knowledge Makes Large Language Models Better In-context
  Learners
Authors: name: Linyi Yang, name: Shuibai Zhang, name: Zhuohao Yu, name: Guangsheng Bao, name: Yidong Wang, name: Jindong Wang, name: Ruochen Xu, name: Wei Ye, name: Xing Xie, name: Weizhu Chen, name: Yue Zhang
Published: 2023-12-26 07:24:46

Large Language Models (LLMs) exhibit emerging in-context learning abilities
through prompt engineering. The recent progress in large-scale generative
models has further expanded their use in real-world language applications.
However, the critical challenge of improving the generalizability and
factuality of LLMs in natural language understanding and question answering
remains under-explored. While previous in-context learning research has focused
on enhancing models to adhere to users' specific instructions and quality
expectations, and to avoid undesired outputs, little to no work has explored
the use of task-Specific fine-tuned Language Models (SLMs) to improve LLMs'
in-context learning during the inference stage. Our primary contribution is the
establishment of a simple yet effective framework that enhances the reliability
of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs
benefit from discriminative models, and 3) minimizes hallucinations in
generative tasks. Using our proposed plug-in method, enhanced versions of Llama
2 and ChatGPT surpass their original versions regarding generalizability and
factuality. We offer a comprehensive suite of resources, including 16 curated
datasets, prompts, model checkpoints, and LLM outputs across 9 distinct tasks.
Our empirical analysis sheds light on the advantages of incorporating
discriminative models into LLMs and highlights the potential of our methodology
in fostering more reliable LLMs.
Gemini vs GPT-4V: A Preliminary Comparison and Combination of
  Vision-Language Models Through Qualitative Cases
Authors: name: Zhangyang Qi, name: Ye Fang, name: Mengchen Zhang, name: Zeyi Sun, name: Tong Wu, name: Ziwei Liu, name: Dahua Lin, name: Jiaqi Wang, name: Hengshuang Zhao
Published: 2023-12-22 18:59:58

The rapidly evolving sector of Multi-modal Large Language Models (MLLMs) is
at the forefront of integrating linguistic and visual processing in artificial
intelligence. This paper presents an in-depth comparative study of two
pioneering models: Google's Gemini and OpenAI's GPT-4V(ision). Our study
involves a multi-faceted evaluation of both models across key dimensions such
as Vision-Language Capability, Interaction with Humans, Temporal Understanding,
and assessments in both Intelligence and Emotional Quotients. The core of our
analysis delves into the distinct visual comprehension abilities of each model.
We conducted a series of structured experiments to evaluate their performance
in various industrial application scenarios, offering a comprehensive
perspective on their practical utility. We not only involve direct performance
comparisons but also include adjustments in prompts and scenarios to ensure a
balanced and fair analysis. Our findings illuminate the unique strengths and
niches of both models. GPT-4V distinguishes itself with its precision and
succinctness in responses, while Gemini excels in providing detailed, expansive
answers accompanied by relevant imagery and links. These understandings not
only shed light on the comparative merits of Gemini and GPT-4V but also
underscore the evolving landscape of multimodal foundation models, paving the
way for future advancements in this area. After the comparison, we attempted to
achieve better results by combining the two models. Finally, We would like to
express our profound gratitude to the teams behind GPT-4V and Gemini for their
pioneering contributions to the field. Our acknowledgments are also extended to
the comprehensive qualitative analysis presented in 'Dawn' by Yang et al. This
work, with its extensive collection of image samples, prompts, and
GPT-4V-related results, provided a foundational basis for our analysis.
Parrot Captions Teach CLIP to Spot Text
Authors: name: Yiqi Lin, name: Conghui He, name: Alex Jinpeng Wang, name: Bin Wang, name: Weijia Li, name: Mike Zheng Shou
Published: 2023-12-21 18:46:46

Despite CLIP being the foundation model in numerous vision-language
applications, the CLIP suffers from a severe text spotting bias. Such bias
causes CLIP models to `Parrot' the visual text embedded within images while
disregarding the authentic visual semantics. We uncover that in the most
popular image-text dataset LAION-2B, the captions also densely parrot (spell)
the text embedded in images. Our analysis shows that around 50\% of
images are embedded with visual text content, and 90\% of their
captions more or less parrot the visual text. Based on such observation, we
thoroughly inspect the different release d versions of CLIP models and verify
that the visual text is the dominant factor in measuring the LAION-style
image-text similarity for these models. To examine whether these parrot
captions shape the text spotting bias, we train a series of CLIP models with
LAION subsets curated by different parrot-caption-oriented criteria. We show
that training with parrot captions easily shapes such bias but harms the
expected visual-language representation learning in CLIP models. This suggests
that it is urgent to revisit either the design of CLIP-like models or the
existing image-text dataset curation pipeline built on CLIP score filtering.
ZeroShape: Regression-based Zero-shot Shape Reconstruction
Authors: name: Zixuan Huang, name: Stefan Stojanov, name: Anh Thai, name: Varun Jampani, name: James M. Rehg
Published: 2023-12-21 01:56:34

We study the problem of single-image zero-shot 3D shape reconstruction.
Recent works learn zero-shot shape reconstruction through generative modeling
of 3D assets, but these models are computationally expensive at train and
inference time. In contrast, the traditional approach to this problem is
regression-based, where deterministic models are trained to directly regress
the object shape. Such regression methods possess much higher computational
efficiency than generative methods. This raises a natural question: is
generative modeling necessary for high performance, or conversely, are
regression-based approaches still competitive? To answer this, we design a
strong regression-based model, called ZeroShape, based on the converging
findings in this field and a novel insight. We also curate a large real-world
evaluation benchmark, with objects from three different real-world 3D datasets.
This evaluation benchmark is more diverse and an order of magnitude larger than
what prior works use to quantitatively evaluate their models, aiming at
reducing the evaluation variance in our field. We show that ZeroShape not only
achieves superior performance over state-of-the-art methods, but also
demonstrates significantly higher computational and data efficiency.
MACS: Mass Conditioned 3D Hand and Object Motion Synthesis
Authors: name: Soshi Shimada, name: Franziska Mueller, name: Jan Bednarik, name: Bardia Doosti, name: Bernd Bickel, name: Danhang Tang, name: Vladislav Golyanik, name: Jonathan Taylor, name: Christian Theobalt, name: Thabo Beeler
Published: 2023-12-22 18:59:54

The physical properties of an object, such as mass, significantly affect how
we manipulate it with our hands. Surprisingly, this aspect has so far been
neglected in prior work on 3D motion synthesis. To improve the naturalness of
the synthesized 3D hand object motions, this work proposes MACS the first MAss
Conditioned 3D hand and object motion Synthesis approach. Our approach is based
on cascaded diffusion models and generates interactions that plausibly adjust
based on the object mass and interaction type. MACS also accepts a manually
drawn 3D object trajectory as input and synthesizes the natural 3D hand motions
conditioned by the object mass. This flexibility enables MACS to be used for
various downstream applications, such as generating synthetic training data for
ML tasks, fast animation of hands for graphics workflows, and generating
character interactions for computer games. We show experimentally that a
small-scale dataset is sufficient for MACS to reasonably generalize across
interpolated and extrapolated object masses unseen during the training.
Furthermore, MACS shows moderate generalization to unseen objects, thanks to
the mass-conditioned contact labels generated by our surface contact synthesis
model ConNet. Our comprehensive user study confirms that the synthesized 3D
hand-object interactions are highly plausible and realistic.
Generative AI Beyond LLMs: System Implications of Multi-Modal Generation
Authors: name: Alicia Golden, name: Samuel Hsia, name: Fei Sun, name: Bilge Acun, name: Basil Hosmer, name: Yejin Lee, name: Zachary DeVito, name: Jeff Johnson, name: Gu-Yeon Wei, name: David Brooks, name: Carole-Jean Wu
Published: 2023-12-22 02:21:26

As the development of large-scale Generative AI models evolve beyond text
(1D) generation to include image (2D) and video (3D) generation, processing
spatial and temporal information presents unique challenges to quality,
performance, and efficiency. We present the first work towards understanding
this new system design space for multi-modal text-to-image (TTI) and
text-to-video (TTV) generation models. Current model architecture designs are
bifurcated into 2 categories: Diffusion- and Transformer-based models. Our
systematic performance characterization on a suite of eight representative
TTI/TTV models shows that after state-of-the-art optimization techniques such
as Flash Attention are applied, Convolution accounts for up to 44% of execution
time for Diffusion-based TTI models, while Linear layers consume up to 49% of
execution time for Transformer-based models. We additionally observe that
Diffusion-based TTI models resemble the Prefill stage of LLM inference, and
benefit from 1.1-2.5x greater speedup from Flash Attention than
Transformer-based TTI models that resemble the Decode phase. Since
optimizations designed for LLMs do not map directly onto TTI/TTV models, we
must conduct a thorough characterization of these workloads to gain insights
for new optimization opportunities. In doing so, we define sequence length in
the context of TTI/TTV models and observe sequence length can vary up to 4x in
Diffusion model inference. We additionally observe temporal aspects of TTV
workloads pose unique system bottlenecks, with Temporal Attention accounting
for over 60% of total Attention time. Overall, our in-depth system performance
characterization is a critical first step towards designing efficient and
deployable systems for emerging TTI/TTV workloads.
DreamDistribution: Prompt Distribution Learning for Text-to-Image
  Diffusion Models
Authors: name: Brian Nlong Zhao, name: Yuhang Xiao, name: Jiashu Xu, name: Xinyang Jiang, name: Yifan Yang, name: Dongsheng Li, name: Laurent Itti, name: Vibhav Vineet, name: Yunhao Ge
Published: 2023-12-21 12:11:00

The popularization of Text-to-Image (T2I) diffusion models enables the
generation of high-quality images from text descriptions. However, generating
diverse customized images with reference visual attributes remains challenging.
This work focuses on personalizing T2I diffusion models at a more abstract
concept or category level, adapting commonalities from a set of reference
images while creating new instances with sufficient variations. We introduce a
solution that allows a pretrained T2I diffusion model to learn a set of soft
prompts, enabling the generation of novel images by sampling prompts from the
learned distribution. These prompts offer text-guided editing capabilities and
additional flexibility in controlling variation and mixing between multiple
distributions. We also show the adaptability of the learned prompt distribution
to other tasks, such as text-to-3D. Finally we demonstrate effectiveness of our
approach through quantitative analysis including automatic evaluation and human
assessment. Project website: https://briannlongzhao.github.io/DreamDistribution
PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce
  Lidar
Authors: name: Tzofi Klinghoffer, name: Xiaoyu Xiang, name: Siddharth Somasundaram, name: Yuchen Fan, name: Christian Richardt, name: Ramesh Raskar, name: Rakesh Ranjan
Published: 2023-12-21 18:59:53

3D reconstruction from a single-view is challenging because of the ambiguity
from monocular cues and lack of information about occluded regions. Neural
radiance fields (NeRF), while popular for view synthesis and 3D reconstruction,
are typically reliant on multi-view images. Existing methods for single-view 3D
reconstruction with NeRF rely on either data priors to hallucinate views of
occluded regions, which may not be physically accurate, or shadows observed by
RGB cameras, which are difficult to detect in ambient light and low albedo
backgrounds. We propose using time-of-flight data captured by a single-photon
avalanche diode to overcome these limitations. Our method models two-bounce
optical paths with NeRF, using lidar transient data for supervision. By
leveraging the advantages of both NeRF and two-bounce light measured by lidar,
we demonstrate that we can reconstruct visible and occluded geometry without
data priors or reliance on controlled ambient lighting or scene albedo. In
addition, we demonstrate improved generalization under practical constraints on
sensor spatial- and temporal-resolution. We believe our method is a promising
direction as single-photon lidars become ubiquitous on consumer devices, such
as phones, tablets, and headsets.
Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning
Authors: name: Filippos Christianos, name: Georgios Papoudakis, name: Matthieu Zimmer, name: Thomas Coste, name: Zhihao Wu, name: Jingxuan Chen, name: Khyati Khandelwal, name: James Doran, name: Xidong Feng, name: Jiacheng Liu, name: Zheng Xiong, name: Yicheng Luo, name: Jianye Hao, name: Kun Shao, name: Haitham Bou-Ammar, name: Jun Wang
Published: 2023-12-22 17:57:57

A key method for creating Artificial Intelligence (AI) agents is
Reinforcement Learning (RL). However, constructing a standalone RL policy that
maps perception to action directly encounters severe problems, chief among them
being its lack of generality across multiple tasks and the need for a large
amount of training data. The leading cause is that it cannot effectively
integrate prior information into the perception-action cycle when devising the
policy. Large language models (LLMs) emerged as a fundamental way to
incorporate cross-domain knowledge into AI agents but lack crucial learning and
adaptation toward specific decision problems. This paper presents a general
framework model for integrating and learning structured reasoning into AI
agents' policies. Our methodology is motivated by the modularity found in the
human brain. The framework utilises the construction of intrinsic and extrinsic
functions to add previous understandings of reasoning structures. It also
provides the adaptive ability to learn models inside every module or function,
consistent with the modular structure of cognitive processes. We describe the
framework in-depth and compare it with other AI pipelines and existing
frameworks. The paper explores practical applications, covering experiments
that show the effectiveness of our method. Our results indicate that AI agents
perform and adapt far better when organised reasoning and prior knowledge are
embedded. This opens the door to more resilient and general AI agent systems.
YAYI 2: Multilingual Open-Source Large Language Models
Authors: name: Yin Luo, name: Qingchao Kong, name: Nan Xu, name: Jia Cao, name: Bao Hao, name: Baoyu Qu, name: Bo Chen, name: Chao Zhu, name: Chenyang Zhao, name: Donglei Zhang, name: Fan Feng, name: Feifei Zhao, name: Hailong Sun, name: Hanxuan Yang, name: Haojun Pan, name: Hongyu Liu, name: Jianbin Guo, name: Jiangtao Du, name: Jingyi Wang, name: Junfeng Li, name: Lei Sun, name: Liduo Liu, name: Lifeng Dong, name: Lili Liu, name: Lin Wang, name: Liwen Zhang, name: Minzheng Wang, name: Pin Wang, name: Ping Yu, name: Qingxiao Li, name: Rui Yan, name: Rui Zou, name: Ruiqun Li, name: Taiwen Huang, name: Xiaodong Wang, name: Xiaofei Wu, name: Xin Peng, name: Xina Zhang, name: Xing Fang, name: Xinglin Xiao, name: Yanni Hao, name: Yao Dong, name: Yigang Wang, name: Ying Liu, name: Yongyu Jiang, name: Yungan Wang, name: Yuqi Wang, name: Zhangsheng Wang, name: Zhaoxin Yu, name: Zhen Luo, name: Wenji Mao, name: Lei Wang, name: Dajun Zeng
Published: 2023-12-22 17:34:47

As the latest advancements in natural language processing, large language
models (LLMs) have achieved human-level language understanding and generation
abilities in many real-world tasks, and even have been regarded as a potential
path to the artificial general intelligence. To better facilitate research on
LLMs, many open-source LLMs, such as Llama 2 and Falcon, have recently been
proposed and gained comparable performances to proprietary models. However,
these models are primarily designed for English scenarios and exhibit poor
performances in Chinese contexts. In this technical report, we propose YAYI 2,
including both base and chat models, with 30 billion parameters. YAYI 2 is
pre-trained from scratch on a multilingual corpus which contains 2.65 trillion
tokens filtered by our pre-training data processing pipeline. The base model is
aligned with human values through supervised fine-tuning with millions of
instructions and reinforcement learning from human feedback. Extensive
experiments on multiple benchmarks, such as MMLU and CMMLU, consistently
demonstrate that the proposed YAYI 2 outperforms other similar sized
open-source models.
Reasons to Reject? Aligning Language Models with Judgments
Authors: name: Weiwen Xu, name: Deng Cai, name: Zhisong Zhang, name: Wai Lam, name: Shuming Shi
Published: 2023-12-22 10:29:43

As humans, we consistently engage in interactions with our peers and receive
feedback in the form of natural language. This language feedback allows us to
reflect on our actions, maintain appropriate behavior, and rectify our errors.
The question arises naturally: can we use language feedback to align large
language models (LLMs)? In contrast to previous research that aligns LLMs with
reward or preference data, we present the first systematic exploration of
alignment through the lens of language feedback (i.e., judgment). We commence
with an in-depth investigation of potential methods that can be adapted for
aligning LLMs with judgments, revealing that these methods are unable to fully
capitalize on the judgments. To facilitate more effective utilization of
judgments, we propose a novel framework, Contrastive Unlikelihood Training
(CUT), that allows for fine-grained inappropriate content detection and
correction based on judgments. Our offline alignment results show that, with
merely 1317 off-the-shelf judgment data, CUT (LLaMA2-13b) can beat the 175B
DaVinci003 and surpass the best baseline by 52.34 points on AlpacaEval. The
online alignment results demonstrate that CUT can align LLMs (LLaMA2-chat-13b)
in an iterative fashion using model-specific judgment data, with a steady
performance improvement from 81.09 to 91.36 points on AlpacaEval. Our analysis
further suggests that judgments exhibit greater potential than rewards for LLM
alignment and warrant future research.
Parameter Efficient Tuning Allows Scalable Personalization of LLMs for
  Text Entry: A Case Study on Abbreviation Expansion
Authors: name: Katrin Tomanek, name: Shanqing Cai, name: Subhashini Venugopalan
Published: 2023-12-21 22:52:44

Abbreviation expansion is a strategy used to speed up communication by
limiting the amount of typing and using a language model to suggest expansions.
Here we look at personalizing a Large Language Model's (LLM) suggestions based
on prior conversations to enhance the relevance of predictions, particularly
when the user data is small (~1000 samples). Specifically, we compare
fine-tuning, prompt-tuning, and retrieval augmented generation of expanded text
suggestions for abbreviated inputs. Our case study with a deployed 8B parameter
LLM on a real user living with ALS, and experiments on movie character
personalization indicates that (1) customization may be necessary in some
scenarios and prompt-tuning generalizes well to those, (2) fine-tuning on
in-domain data (with as few as 600 samples) still shows some gains, however (3)
retrieval augmented few-shot selection also outperforms fine-tuning. (4)
Parameter efficient tuning allows for efficient and scalable personalization.
For prompt-tuning, we also find that initializing the learned "soft-prompts" to
user relevant concept tokens leads to higher accuracy than random
initialization.