Orionnnn/hub-stats huggingface dataset:
Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective
  Distillation and Unlabeled Data Augmentation
Authors: name: Ning-Hsu Wang, name: Yu-Lun Liu
Published: 2024-06-18 17:59:31

Accurately estimating depth in 360-degree imagery is crucial for virtual
reality, autonomous navigation, and immersive media applications. Existing
depth estimation methods designed for perspective-view imagery fail when
applied to 360-degree images due to different camera projections and
distortions, whereas 360-degree methods perform inferior due to the lack of
labeled data pairs. We propose a new depth estimation framework that utilizes
unlabeled 360-degree data effectively. Our approach uses state-of-the-art
perspective depth estimation models as teacher models to generate pseudo labels
through a six-face cube projection technique, enabling efficient labeling of
depth in 360-degree images. This method leverages the increasing availability
of large datasets. Our approach includes two main stages: offline mask
generation for invalid regions and an online semi-supervised joint training
regime. We tested our approach on benchmark datasets such as Matterport3D and
Stanford2D3D, showing significant improvements in depth estimation accuracy,
particularly in zero-shot scenarios. Our proposed training pipeline can enhance
any 360 monocular depth estimator and demonstrates effective knowledge transfer
across different camera projections and data types. See our project page for
results: https://albert100121.github.io/Depth-Anywhere/
AgileCoder: Dynamic Collaborative Agents for Software Development based
  on Agile Methodology
Authors: name: Minh Huynh Nguyen, name: Thang Phan Chau, name: Phong X. Nguyen, name: Nghi D. Q. Bui
Published: 2024-06-16 17:57:48

Software agents have emerged as promising tools for addressing complex
software engineering tasks. However, existing works oversimplify software
development workflows by following the waterfall model. Thus, we propose
AgileCoder, a multi-agent system that integrates Agile Methodology (AM) into
the framework. This system assigns specific AM roles such as Product Manager,
Developer, and Tester to different agents, who then collaboratively develop
software based on user inputs. AgileCoder enhances development efficiency by
organizing work into sprints, focusing on incrementally developing software
through sprints. Additionally, we introduce Dynamic Code Graph Generator, a
module that creates a Code Dependency Graph dynamically as updates are made to
the codebase. This allows agents to better comprehend the codebase, leading to
more precise code generation and modifications throughout the software
development process. AgileCoder surpasses existing benchmarks, like ChatDev and
MetaGPT, establishing a new standard and showcasing the capabilities of
multi-agent systems in advanced software engineering environments. Our source
code can be found at https://github.com/FSoft-AI4Code/AgileCoder.
Large Scale Transfer Learning for Tabular Data via Language Modeling
Authors: name: Josh Gardner, name: Juan C. Perdomo, name: Ludwig Schmidt
Published: 2024-06-17 18:58:20

Tabular data -- structured, heterogeneous, spreadsheet-style data with rows
and columns -- is widely used in practice across many domains. However, while
recent foundation models have reduced the need for developing task-specific
datasets and predictors in domains such as language modeling and computer
vision, this transfer learning paradigm has not had similar impact in the
tabular domain. In this work, we seek to narrow this gap and present TabuLa-8B,
a language model for tabular prediction. We define a process for extracting a
large, high-quality training dataset from the TabLib corpus, proposing methods
for tabular data filtering and quality control. Using the resulting dataset,
which comprises over 1.6B rows from 3.1M unique tables, we fine-tune a Llama
3-8B large language model (LLM) for tabular data prediction (classification and
binned regression) using a novel packing and attention scheme for tabular
prediction. Through evaluation across a test suite of 329 datasets, we find
that TabuLa-8B has zero-shot accuracy on unseen tables that is over 15
percentage points (pp) higher than random guessing, a feat that is not possible
with existing state-of-the-art tabular prediction models (e.g. XGBoost,
TabPFN). In the few-shot setting (1-32 shots), without any fine-tuning on the
target datasets, TabuLa-8B is 5-15 pp more accurate than XGBoost and TabPFN
models that are explicitly trained on equal, or even up to 16x more data. We
release our model, code, and data along with the publication of this paper.
HumanSplat: Generalizable Single-Image Human Gaussian Splatting with
  Structure Priors
Authors: name: Panwang Pan, name: Zhuo Su, name: Chenguo Lin, name: Zhen Fan, name: Yongjie Zhang, name: Zeming Li, name: Tingting Shen, name: Yadong Mu, name: Yebin Liu
Published: 2024-06-18 10:05:33

Despite recent advancements in high-fidelity human reconstruction techniques,
the requirements for densely captured images or time-consuming per-instance
optimization significantly hinder their applications in broader scenarios. To
tackle these issues, we present HumanSplat which predicts the 3D Gaussian
Splatting properties of any human from a single input image in a generalizable
manner. In particular, HumanSplat comprises a 2D multi-view diffusion model and
a latent reconstruction transformer with human structure priors that adeptly
integrate geometric priors and semantic features within a unified framework. A
hierarchical loss that incorporates human semantic information is further
designed to achieve high-fidelity texture modeling and better constrain the
estimated multiple views. Comprehensive experiments on standard benchmarks and
in-the-wild images demonstrate that HumanSplat surpasses existing
state-of-the-art methods in achieving photorealistic novel-view synthesis.
Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for
  Large Language Models
Authors: name: Dongwon Jo, name: Taesu Kim, name: Yulhwa Kim, name: Jae-Joon Kim
Published: 2024-06-18 06:32:23

Binarization, which converts weight parameters to binary values, has emerged
as an effective strategy to reduce the size of large language models (LLMs).
However, typical binarization techniques significantly diminish linguistic
effectiveness of LLMs. To address this issue, we introduce a novel binarization
technique called Mixture of Scales (BinaryMoS). Unlike conventional methods,
BinaryMoS employs multiple scaling experts for binary weights, dynamically
merging these experts for each token to adaptively generate scaling factors.
This token-adaptive approach boosts the representational power of binarized
LLMs by enabling contextual adjustments to the values of binary weights.
Moreover, because this adaptive process only involves the scaling factors
rather than the entire weight matrix, BinaryMoS maintains compression
efficiency similar to traditional static binarization methods. Our experimental
results reveal that BinaryMoS surpasses conventional binarization techniques in
various natural language processing tasks and even outperforms 2-bit
quantization methods, all while maintaining similar model size to static
binarization techniques.
Tokenization Falling Short: The Curse of Tokenization
Authors: name: Yekun Chai, name: Yewei Fang, name: Qiwei Peng, name: Xuhong Li
Published: 2024-06-17 16:05:32

Language models typically tokenize raw text into sequences of subword
identifiers from a predefined vocabulary, a process inherently sensitive to
typographical errors, length variations, and largely oblivious to the internal
structure of tokens-issues we term the curse of tokenization. In this study, we
delve into these drawbacks and demonstrate that large language models (LLMs)
remain susceptible to these problems. This study systematically investigates
these challenges and their impact on LLMs through three critical research
questions: (1) complex problem solving, (2) token structure probing, and (3)
resilience to typographical variation. Our findings reveal that scaling model
parameters can mitigate the issue of tokenization; however, LLMs still suffer
from biases induced by typos and other text format variations. Our experiments
show that subword regularization such as BPE-dropout can mitigate this issue.
We will release our code and data to facilitate further research.
Benchmarking Multi-Image Understanding in Vision and Language Models:
  Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
Authors: name: Bingchen Zhao, name: Yongshuo Zong, name: Letian Zhang, name: Timothy Hospedales
Published: 2024-06-18 16:02:18

The advancement of large language models (LLMs) has significantly broadened
the scope of applications in natural language processing, with multi-modal LLMs
extending these capabilities to integrate and interpret visual data. However,
existing benchmarks for visual language models (VLMs) predominantly focus on
single-image inputs, neglecting the crucial aspect of multi-image
understanding. In this paper, we introduce a Multi-Image Relational Benchmark
MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across
multiple images. Our benchmark encompasses four categories: perception, visual
world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive
evaluation of a wide range of open-source and closed-source models, we
demonstrate that while open-source VLMs were shown to approach the performance
of GPT-4V in single-image tasks, a significant performance gap remains in
multi-image reasoning tasks. Our findings also reveal that even the
state-of-the-art GPT-4V model struggles with our benchmark, underscoring the
need for further research and development in this area. We believe our
contribution of MIRB could serve as a testbed for developing the
next-generation multi-modal models.
Bootstrapping Language Models with DPO Implicit Rewards
Authors: name: Changyu Chen, name: Zichen Liu, name: Chao Du, name: Tianyu Pang, name: Qian Liu, name: Arunesh Sinha, name: Pradeep Varakantham, name: Min Lin
Published: 2024-06-14 06:57:18

Human alignment in large language models (LLMs) is an active area of
research. A recent groundbreaking work, direct preference optimization (DPO),
has greatly simplified the process from past work in reinforcement learning
from human feedback (RLHF) by bypassing the reward learning stage in RLHF. DPO,
after training, provides an implicit reward model. In this work, we make a
novel observation that this implicit reward model can by itself be used in a
bootstrapping fashion to further align the LLM. Our approach is to use the
rewards from a current LLM model to construct a preference dataset, which is
then used in subsequent DPO rounds. We incorporate refinements that debias the
length of the responses and improve the quality of the preference dataset to
further improve our approach. Our approach, named self-alignment with DPO
ImpliCit rEwards (DICE), shows great improvements in alignment and achieves
superior performance than Gemini Pro on AlpacaEval 2, reaching 27.55%
length-controlled win rate against GPT-4 Turbo, but with only 8B parameters and
no external feedback. Our code is available at https://github.com/sail-sg/dice.
BPO: Supercharging Online Preference Learning by Adhering to the
  Proximity of Behavior LLM
Authors: name: Wenda Xu, name: Jiachen Li, name: William Yang Wang, name: Lei Li
Published: 2024-06-18 00:41:40

Direct alignment from preferences (DAP) has emerged as a promising paradigm
for aligning large language models (LLMs) to human desiderata from
pre-collected, offline preference datasets. While recent studies indicate that
existing offline DAP methods can directly benefit from online training samples,
we highlight the need to develop specific online DAP algorithms to fully
harness the power of online training. Specifically, we identify that the
learned LLM should adhere to the proximity of the behavior LLM, which collects
the training samples. To this end, we propose online Preference Optimization in
proximity to the Behavior LLM (BPO), emphasizing the importance of constructing
a proper trust region for LLM alignment.
  We conduct extensive experiments to validate the effectiveness and
applicability of our approach by integrating it with various DAP methods,
resulting in significant performance improvements across a wide range of tasks
when training with the same amount of preference data. Even when only
introducing one additional data collection phase, our online BPO improves its
offline DAP baseline from 72.0% to 80.2% on TL;DR and from 82.2% to 89.1% on
Anthropic Helpfulness in terms of win rate against human reference text.
Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for
  Large Language Models
Authors: name: Devichand Budagam, name: Sankalp KJ, name: Ashutosh Kumar, name: Vinija Jain, name: Aman Chadha
Published: 2024-06-18 14:12:27

Assessing the effectiveness of large language models (LLMs) in addressing
diverse tasks is essential for comprehending their strengths and weaknesses.
Conventional evaluation techniques typically apply a single prompting strategy
uniformly across datasets, not considering the varying degrees of task
complexity. We introduce the Hierarchical Prompting Taxonomy (HPT), a taxonomy
that employs a Hierarchical Prompt Framework (HPF) composed of five unique
prompting strategies, arranged from the simplest to the most complex, to assess
LLMs more precisely and to offer a clearer perspective. This taxonomy assigns a
score, called the Hierarchical Prompting Score (HP-Score), to datasets as well
as LLMs based on the rules of the taxonomy, providing a nuanced understanding
of their ability to solve diverse tasks and offering a universal measure of
task complexity. Additionally, we introduce the Adaptive Hierarchical Prompt
framework, which automates the selection of appropriate prompting strategies
for each task. This study compares manual and adaptive hierarchical prompt
frameworks using four instruction-tuned LLMs, namely Llama 3 8B, Phi 3 3.8B,
Mistral 7B, and Gemma 7B, across four datasets: BoolQ, CommonSenseQA (CSQA),
IWSLT-2017 en-fr (IWSLT), and SamSum. Experiments demonstrate the effectiveness
of HPT, providing a reliable way to compare different tasks and LLM
capabilities. This paper leads to the development of a universal evaluation
metric that can be used to evaluate both the complexity of the datasets and the
capabilities of LLMs. The implementation of both manual HPF and adaptive HPF is
publicly available.
JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal
  Parameters Tuning
Authors: name: Boyu Chen, name: Peike Li, name: Yao Yao, name: Alex Wang
Published: 2024-06-18 05:54:11

Large models for text-to-music generation have achieved significant progress,
facilitating the creation of high-quality and varied musical compositions from
provided text prompts. However, input text prompts may not precisely capture
user requirements, particularly when the objective is to generate music that
embodies a specific concept derived from a designated reference collection. In
this paper, we propose a novel method for customized text-to-music generation,
which can capture the concept from a two-minute reference music and generate a
new piece of music conforming to the concept. We achieve this by fine-tuning a
pretrained text-to-music model using the reference music. However, directly
fine-tuning all parameters leads to overfitting issues. To address this
problem, we propose a Pivotal Parameters Tuning method that enables the model
to assimilate the new concept while preserving its original generative
capabilities. Additionally, we identify a potential concept conflict when
introducing multiple concepts into the pretrained model. We present a concept
enhancement strategy to distinguish multiple concepts, enabling the fine-tuned
model to generate music incorporating either individual or multiple concepts
simultaneously. Since we are the first to work on the customized music
generation task, we also introduce a new dataset and evaluation protocol for
the new task. Our proposed Jen1-DreamStyler outperforms several baselines in
both qualitative and quantitative evaluations. Demos will be available at
https://www.jenmusic.ai/research#DreamStyler.
From RAGs to rich parameters: Probing how language models utilize
  external knowledge over parametric information for factual queries
Authors: name: Hitesh Wadhwa, name: Rahul Seetharaman, name: Somyaa Aggarwal, name: Reshmi Ghosh, name: Samyadeep Basu, name: Soundararajan Srinivasan, name: Wenlong Zhao, name: Shreyas Chaudhari, name: Ehsan Aghazadeh
Published: 2024-06-18 17:46:08

Retrieval Augmented Generation (RAG) enriches the ability of language models
to reason using external context to augment responses for a given user prompt.
This approach has risen in popularity due to practical applications in various
applications of language models in search, question/answering, and chat-bots.
However, the exact nature of how this approach works isn't clearly understood.
In this paper, we mechanistically examine the RAG pipeline to highlight that
language models take shortcut and have a strong bias towards utilizing only the
context information to answer the question, while relying minimally on their
parametric memory. We probe this mechanistic behavior in language models with:
(i) Causal Mediation Analysis to show that the parametric memory is minimally
utilized when answering a question and (ii) Attention Contributions and
Knockouts to show that the last token residual stream do not get enriched from
the subject token in the question, but gets enriched from other informative
tokens in the context. We find this pronounced shortcut behaviour true across
both LLaMa and Phi family of models.
VoCo-LLaMA: Towards Vision Compression with Large Language Models
Authors: name: Xubing Ye, name: Yukang Gan, name: Xiaoke Huang, name: Yixiao Ge, name: Ying Shan, name: Yansong Tang
Published: 2024-06-18 05:05:12

Vision-Language Models (VLMs) have achieved remarkable success in various
multi-modal tasks, but they are often bottlenecked by the limited context
window and high computational cost of processing high-resolution image inputs
and videos. Vision compression can alleviate this problem by reducing the
vision token count. Previous approaches compress vision tokens with external
modules and force LLMs to understand the compressed ones, leading to visual
information loss. However, the LLMs' understanding paradigm of vision tokens is
not fully utilised in the compression learning process. We propose VoCo-LLaMA,
the first approach to compress vision tokens using LLMs. By introducing Vision
Compression tokens during the vision instruction tuning phase and leveraging
attention distillation, our method distill how LLMs comprehend vision tokens
into their processing of VoCo tokens. VoCo-LLaMA facilitates effective vision
compression and improves the computational efficiency during the inference
stage. Specifically, our method achieves minimal performance loss with a
compression ratio of 576times, resulting in up to 94.8% fewer FLOPs and
69.6% acceleration in inference time. Furthermore, through continuous
training using time-series compressed token sequences of video frames,
VoCo-LLaMA demonstrates the ability to understand temporal correlations,
outperforming previous methods on popular video question-answering benchmarks.
Our approach presents a promising way to unlock the full potential of VLMs'
contextual window, enabling more scalable multi-modal applications. The project
page, along with the associated code, can be accessed via
https://yxxxb.github.io/VoCo-LLaMA-page/{this https URL}.
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All
  Tools
Authors: name: Team GLM, name: Aohan Zeng, name: Bin Xu, name: Bowen Wang, name: Chenhui Zhang, name: Da Yin, name: Diego Rojas, name: Guanyu Feng, name: Hanlin Zhao, name: Hanyu Lai, name: Hao Yu, name: Hongning Wang, name: Jiadai Sun, name: Jiajie Zhang, name: Jiale Cheng, name: Jiayi Gui, name: Jie Tang, name: Jing Zhang, name: Juanzi Li, name: Lei Zhao, name: Lindong Wu, name: Lucen Zhong, name: Mingdao Liu, name: Minlie Huang, name: Peng Zhang, name: Qinkai Zheng, name: Rui Lu, name: Shuaiqi Duan, name: Shudan Zhang, name: Shulin Cao, name: Shuxun Yang, name: Weng Lam Tam, name: Wenyi Zhao, name: Xiao Liu, name: Xiao Xia, name: Xiaohan Zhang, name: Xiaotao Gu, name: Xin Lv, name: Xinghan Liu, name: Xinyi Liu, name: Xinyue Yang, name: Xixuan Song, name: Xunkai Zhang, name: Yifan An, name: Yifan Xu, name: Yilin Niu, name: Yuantao Yang, name: Yueyan Li, name: Yushi Bai, name: Yuxiao Dong, name: Zehan Qi, name: Zhaoyu Wang, name: Zhen Yang, name: Zhengxiao Du, name: Zhenyu Hou, name: Zihan Wang
Published: 2024-06-18 16:58:21

We introduce ChatGLM, an evolving family of large language models that we
have been developing over time. This report primarily focuses on the GLM-4
language series, which includes GLM-4, GLM-4-Air, and GLM-4-9B. They represent
our most capable models that are trained with all the insights and lessons
gained from the preceding three generations of ChatGLM. To date, the GLM-4
models are pre-trained on ten trillions of tokens mostly in Chinese and
English, along with a small set of corpus from 24 languages, and aligned
primarily for Chinese and English usage. The high-quality alignment is achieved
via a multi-stage post-training process, which involves supervised fine-tuning
and learning from human feedback. Evaluations show that GLM-4 1) closely rivals
or outperforms GPT-4 in terms of general metrics such as MMLU, GSM8K, MATH,
BBH, GPQA, and HumanEval, 2) gets close to GPT-4-Turbo in instruction following
as measured by IFEval, 3) matches GPT-4 Turbo (128K) and Claude 3 for long
context tasks, and 4) outperforms GPT-4 in Chinese alignments as measured by
AlignBench. The GLM-4 All Tools model is further aligned to understand user
intent and autonomously decide when and which tool(s) touse -- including web
browser, Python interpreter, text-to-image model, and user-defined functions --
to effectively complete complex tasks. In practical applications, it matches
and even surpasses GPT-4 All Tools in tasks like accessing online information
via web browsing and solving math problems using Python interpreter. Over the
course, we have open-sourced a series of models, including ChatGLM-6B (three
generations), GLM-4-9B (128K, 1M), GLM-4V-9B, WebGLM, and CodeGeeX, attracting
over 10 million downloads on Hugging face in the year 2023 alone. The open
models can be accessed through https://github.com/THUDM and
https://huggingface.co/THUDM.