Orionnnn/hub-stats huggingface dataset:
From Words to Numbers: Your Large Language Model Is Secretly A Capable
  Regressor When Given In-Context Examples
Authors: name: Robert Vacareanu, name: Vlad-Andrei Negru, name: Vasile Suciu, name: Mihai Surdeanu
Published: 2024-04-11 08:12:43

We analyze how well pre-trained large language models (e.g., Llama2, GPT-4,
Claude 3, etc) can do linear and non-linear regression when given in-context
examples, without any additional training or gradient updates. Our findings
reveal that several large language models (e.g., GPT-4, Claude 3) are able to
perform regression tasks with a performance rivaling (or even outperforming)
that of traditional supervised methods such as Random Forest, Bagging, or
Gradient Boosting. For example, on the challenging Friedman #2 regression
dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM,
Random Forest, KNN, or Gradient Boosting. We then investigate how well the
performance of large language models scales with the number of in-context
exemplars. We borrow from the notion of regret from online learning and
empirically show that LLMs are capable of obtaining a sub-linear regret.
DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic
  Gaussian Splatting
Authors: name: Shijie Zhou, name: Zhiwen Fan, name: Dejia Xu, name: Haoran Chang, name: Pradyumna Chari, name: Tejas Bharadwaj, name: Suya You, name: Zhangyang Wang, name: Achuta Kadambi
Published: 2024-04-10 10:46:59

The increasing demand for virtual reality applications has highlighted the
significance of crafting immersive 3D assets. We present a text-to-3D
360^{circ} scene generation pipeline that facilitates the creation of
comprehensive 360^{circ} scenes for in-the-wild environments in a matter of
minutes. Our approach utilizes the generative power of a 2D diffusion model and
prompt self-refinement to create a high-quality and globally coherent panoramic
image. This image acts as a preliminary "flat" (2D) scene representation.
Subsequently, it is lifted into 3D Gaussians, employing splatting techniques to
enable real-time exploration. To produce consistent 3D geometry, our pipeline
constructs a spatially coherent structure by aligning the 2D monocular depth
into a globally optimized point cloud. This point cloud serves as the initial
state for the centroids of 3D Gaussians. In order to address invisible issues
inherent in single-view inputs, we impose semantic and geometric constraints on
both synthesized and input camera views as regularizations. These guide the
optimization of Gaussians, aiding in the reconstruction of unseen regions. In
summary, our method offers a globally consistent 3D scene within a
360^{circ} perspective, providing an enhanced immersive experience over
existing techniques. Project website at: http://dreamscene360.github.io/
BRAVE: Broadening the visual encoding of vision-language models
Authors: name: Oğuzhan Fatih Kar, name: Alessio Tonioni, name: Petra Poklukar, name: Achin Kulshrestha, name: Amir Zamir, name: Federico Tombari
Published: 2024-04-10 17:59:45

Vision-language models (VLMs) are typically composed of a vision encoder,
e.g. CLIP, and a language model (LM) that interprets the encoded features to
solve downstream tasks. Despite remarkable progress, VLMs are subject to
several shortcomings due to the limited capabilities of vision encoders, e.g.
"blindness" to certain image features, visual hallucination, etc. To address
these issues, we study broadening the visual encoding capabilities of VLMs. We
first comprehensively benchmark several vision encoders with different
inductive biases for solving VLM tasks. We observe that there is no single
encoding configuration that consistently achieves top performance across
different tasks, and encoders with different biases can perform surprisingly
similarly. Motivated by this, we introduce a method, named BRAVE, that
consolidates features from multiple frozen encoders into a more versatile
representation that can be directly fed as the input to a frozen LM. BRAVE
achieves state-of-the-art performance on a broad range of captioning and VQA
benchmarks and significantly reduces the aforementioned issues of VLMs, while
requiring a smaller number of trainable parameters than existing methods and
having a more compressed representation. Our results highlight the potential of
incorporating different visual biases for a more broad and contextualized
visual understanding of VLMs.
Adapting LLaMA Decoder to Vision Transformer
Authors: name: Jiahao Wang, name: Wenqi Shao, name: Mengzhao Chen, name: Chengyue Wu, name: Yong Liu, name: Kaipeng Zhang, name: Songyang Zhang, name: Kai Chen, name: Ping Luo
Published: 2024-04-10 06:30:08

This work examines whether decoder-only Transformers such as LLaMA, which
were originally designed for large language models (LLMs), can be adapted to
the computer vision field. We first "LLaMAfy" a standard ViT step-by-step to
align with LLaMA's architecture, and find that directly applying a casual mask
to the self-attention brings an attention collapse issue, resulting in the
failure to the network training. We suggest to reposition the class token
behind the image tokens with a post-sequence class token technique to overcome
this challenge, enabling causal self-attention to efficiently capture the
entire image's information. Additionally, we develop a soft mask strategy that
gradually introduces a casual mask to the self-attention at the onset of
training to facilitate the optimization behavior. The tailored model, dubbed as
image LLaMA (iLLaMA), is akin to LLaMA in architecture and enables direct
supervised learning. Its causal self-attention boosts computational efficiency
and learns complex representation by elevating attention map ranks. iLLaMA
rivals the performance with its encoder-only counterparts, achieving 75.1%
ImageNet top-1 accuracy with only 5.7M parameters. Scaling the model to ~310M
and pre-training on ImageNet-21K further enhances the accuracy to 86.0%.
Extensive experiments demonstrate iLLaMA's reliable properties: calibration,
shape-texture bias, quantization compatibility, ADE20K segmentation and CIFAR
transfer learning. We hope our study can kindle fresh views to visual model
design in the wave of LLMs. Pre-trained models and codes are available here.
RULER: What's the Real Context Size of Your Long-Context Language
  Models?
Authors: name: Cheng-Ping Hsieh, name: Simeng Sun, name: Samuel Kriman, name: Shantanu Acharya, name: Dima Rekesh, name: Fei Jia, name: Boris Ginsburg
Published: 2024-04-09 23:41:27

The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve
a piece of information (the "needle") from long distractor texts (the
"haystack"), has been widely adopted to evaluate long-context language models
(LMs). However, this simple retrieval-based test is indicative of only a
superficial form of long-context understanding. To provide a more comprehensive
evaluation of long-context LMs, we create a new synthetic benchmark RULER with
flexible configurations for customized sequence length and task complexity.
RULER expands upon the vanilla NIAH test to encompass variations with diverse
types and quantities of needles. Moreover, RULER introduces new task categories
multi-hop tracing and aggregation to test behaviors beyond searching from
context. We evaluate ten long-context LMs with 13 representative tasks in
RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, all
models exhibit large performance drops as the context length increases. While
these models all claim context sizes of 32K tokens or greater, only four models
(GPT-4, Command-R, Yi-34B, and Mixtral) can maintain satisfactory performance
at the length of 32K. Our analysis of Yi-34B, which supports context length of
200K, reveals large room for improvement as we increase input length and task
complexity. We open source RULER to spur comprehensive evaluation of
long-context LMs.
RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth
  Diffusion
Authors: name: Jaidev Shriram, name: Alex Trevithick, name: Lingjie Liu, name: Ravi Ramamoorthi
Published: 2024-04-10 17:57:41

We introduce RealmDreamer, a technique for generation of general
forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D
Gaussian Splatting representation to match complex text prompts. We initialize
these splats by utilizing the state-of-the-art text-to-image generators,
lifting their samples into 3D, and computing the occlusion volume. We then
optimize this representation across multiple views as a 3D inpainting task with
image-conditional diffusion models. To learn correct geometric structure, we
incorporate a depth diffusion model by conditioning on the samples from the
inpainting model, giving rich geometric structure. Finally, we finetune the
model using sharpened samples from image generators. Notably, our technique
does not require video or multi-view data and can synthesize a variety of
high-quality 3D scenes in different styles, consisting of multiple objects. Its
generality additionally allows 3D synthesis from a single image.
Leave No Context Behind: Efficient Infinite Context Transformers with
  Infini-attention
Authors: name: Tsendsuren Munkhdalai, name: Manaal Faruqui, name: Siddharth Gopal
Published: 2024-04-10 16:18:42

This work introduces an efficient method to scale Transformer-based Large
Language Models (LLMs) to infinitely long inputs with bounded memory and
computation. A key component in our proposed approach is a new attention
technique dubbed Infini-attention. The Infini-attention incorporates a
compressive memory into the vanilla attention mechanism and builds in both
masked local attention and long-term linear attention mechanisms in a single
Transformer block. We demonstrate the effectiveness of our approach on
long-context language modeling benchmarks, 1M sequence length passkey context
block retrieval and 500K length book summarization tasks with 1B and 8B LLMs.
Our approach introduces minimal bounded memory parameters and enables fast
streaming inference for LLMs.
Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior
Authors: name: Fan Lu, name: Kwan-Yee Lin, name: Yan Xu, name: Hongsheng Li, name: Guang Chen, name: Changjun Jiang
Published: 2024-04-10 06:41:30

Text-to-3D generation has achieved remarkable success via large-scale
text-to-image diffusion models. Nevertheless, there is no paradigm for scaling
up the methodology to urban scale. Urban scenes, characterized by numerous
elements, intricate arrangement relationships, and vast scale, present a
formidable barrier to the interpretability of ambiguous textual descriptions
for effective model optimization. In this work, we surmount the limitations by
introducing a compositional 3D layout representation into text-to-3D paradigm,
serving as an additional prior. It comprises a set of semantic primitives with
simple geometric structures and explicit arrangement relationships,
complementing textual descriptions and enabling steerable generation. Upon
this, we propose two modifications -- (1) We introduce Layout-Guided
Variational Score Distillation to address model optimization inadequacies. It
conditions the score distillation sampling process with geometric and semantic
constraints of 3D layouts. (2) To handle the unbounded nature of urban scenes,
we represent 3D scene with a Scalable Hash Grid structure, incrementally
adapting to the growing scale of urban scenes. Extensive experiments
substantiate the capability of our framework to scale text-to-3D generation to
large-scale urban scenes that cover over 1000m driving distance for the first
time. We also present various scene editing demonstrations, showing the powers
of steerable urban scene generation. Website: https://urbanarchitect.github.io.
Revising Densification in Gaussian Splatting
Authors: name: Samuel Rota Bulò, name: Lorenzo Porzi, name: Peter Kontschieder
Published: 2024-04-09 08:20:37

In this paper, we address the limitations of Adaptive Density Control (ADC)
in 3D Gaussian Splatting (3DGS), a scene representation method achieving
high-quality, photorealistic results for novel view synthesis. ADC has been
introduced for automatic 3D point primitive management, controlling
densification and pruning, however, with certain limitations in the
densification logic. Our main contribution is a more principled, pixel-error
driven formulation for density control in 3DGS, leveraging an auxiliary,
per-pixel error function as the criterion for densification. We further
introduce a mechanism to control the total number of primitives generated per
scene and correct a bias in the current opacity handling strategy of ADC during
cloning operations. Our approach leads to consistent quality improvements
across a variety of benchmark scenes, without sacrificing the method's
efficiency.
Hash3D: Training-free Acceleration for 3D Generation
Authors: name: Xingyi Yang, name: Xinchao Wang
Published: 2024-04-09 07:49:30

The evolution of 3D generative modeling has been notably propelled by the
adoption of 2D diffusion models. Despite this progress, the cumbersome
optimization process per se presents a critical hurdle to efficiency. In this
paper, we introduce Hash3D, a universal acceleration for 3D generation without
model training. Central to Hash3D is the insight that feature-map redundancy is
prevalent in images rendered from camera positions and diffusion time-steps in
close proximity. By effectively hashing and reusing these feature maps across
neighboring timesteps and camera angles, Hash3D substantially prevents
redundant calculations, thus accelerating the diffusion model's inference in 3D
generation tasks. We achieve this through an adaptive grid-based hashing.
Surprisingly, this feature-sharing mechanism not only speed up the generation
but also enhances the smoothness and view consistency of the synthesized 3D
objects. Our experiments covering 5 text-to-3D and 3 image-to-3D models,
demonstrate Hash3D's versatility to speed up optimization, enhancing efficiency
by 1.3 to 4 times. Additionally, Hash3D's integration with 3D Gaussian
splatting largely speeds up 3D model creation, reducing text-to-3D processing
to about 10 minutes and image-to-3D conversion to roughly 30 seconds. The
project page is at https://adamdad.github.io/hash3D/.
Reconstructing Hand-Held Objects in 3D
Authors: name: Jane Wu, name: Georgios Pavlakos, name: Georgia Gkioxari, name: Jitendra Malik
Published: 2024-04-09 17:55:41

Objects manipulated by the hand (i.e., manipulanda) are particularly
challenging to reconstruct from in-the-wild RGB images or videos. Not only does
the hand occlude much of the object, but also the object is often only visible
in a small number of image pixels. At the same time, two strong anchors emerge
in this setting: (1) estimated 3D hands help disambiguate the location and
scale of the object, and (2) the set of manipulanda is small relative to all
possible objects. With these insights in mind, we present a scalable paradigm
for handheld object reconstruction that builds on recent breakthroughs in large
language/vision models and 3D object datasets. Our model, MCC-Hand-Object
(MCC-HO), jointly reconstructs hand and object geometry given a single RGB
image and inferred 3D hand as inputs. Subsequently, we use GPT-4(V) to retrieve
a 3D object model that matches the object in the image and rigidly align the
model to the network-inferred geometry; we call this alignment
Retrieval-Augmented Reconstruction (RAR). Experiments demonstrate that MCC-HO
achieves state-of-the-art performance on lab and Internet datasets, and we show
how RAR can be used to automatically obtain 3D labels for in-the-wild images of
hand-object interactions.
Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence
Authors: name: Bo Peng, name: Daniel Goldstein, name: Quentin Anthony, name: Alon Albalak, name: Eric Alcaide, name: Stella Biderman, name: Eugene Cheah, name: Teddy Ferdinan, name: Haowen Hou, name: Przemysław Kazienko, name: Kranthi Kiran GV, name: Jan Kocoń, name: Bartłomiej Koptyra, name: Satyapriya Krishna, name: Ronald McClelland Jr., name: Niklas Muennighoff, name: Fares Obeid, name: Atsushi Saito, name: Guangyu Song, name: Haoqin Tu, name: Stanisław Woźniak, name: Ruichong Zhang, name: Bingchen Zhao, name: Qihang Zhao, name: Peng Zhou, name: Jian Zhu, name: Rui-Jie Zhu
Published: 2024-04-08 22:20:59

We present Eagle (RWKV-5) and Finch (RWKV-6), sequence models improving upon
the RWKV (RWKV-4) architecture. Our architectural design advancements include
multi-headed matrix-valued states and a dynamic recurrence mechanism that
improve expressivity while maintaining the inference efficiency characteristics
of RNNs. We introduce a new multilingual corpus with 1.12 trillion tokens and a
fast tokenizer based on greedy matching for enhanced multilinguality. We
trained four Eagle models, ranging from 0.46 to 7.5 billion parameters, and two
Finch models with 1.6 and 3.1 billion parameters and find that they achieve
competitive performance across a wide variety of benchmarks. We release all our
models on HuggingFace under the Apache 2.0 license. Models at:
https://huggingface.co/RWKV Training code at: https://github.com/RWKV/RWKV-LM
Inference code at: https://github.com/RWKV/ChatRWKV Time-parallel training code
at: https://github.com/RWKV/RWKV-infctx-trainer
MiniCPM: Unveiling the Potential of Small Language Models with Scalable
  Training Strategies
Authors: name: Shengding Hu, name: Yuge Tu, name: Xu Han, name: Chaoqun He, name: Ganqu Cui, name: Xiang Long, name: Zhi Zheng, name: Yewei Fang, name: Yuxiang Huang, name: Weilin Zhao, name: Xinrong Zhang, name: Zheng Leng Thai, name: Kaihuo Zhang, name: Chongyi Wang, name: Yuan Yao, name: Chenyang Zhao, name: Jie Zhou, name: Jie Cai, name: Zhongwu Zhai, name: Ning Ding, name: Chao Jia, name: Guoyang Zeng, name: Dahai Li, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-04-09 15:36:50

The burgeoning interest in developing Large Language Models (LLMs) with up to
trillion parameters has been met with concerns regarding resource efficiency
and practical expense, particularly given the immense cost of experimentation.
This scenario underscores the importance of exploring the potential of Small
Language Models (SLMs) as a resource-efficient alternative. In this context, we
introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter
variants, not only excel in their respective categories but also demonstrate
capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach
exhibits scalability in both model and data dimensions for future LLM research.
Regarding model scaling, we employ extensive model wind tunnel experiments for
stable and optimal scaling. For data scaling, we introduce a
Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to
continuous training and domain adaptation. We present an in-depth analysis of
the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we
are now able to efficiently study data-model scaling law without extensive
retraining experiments on both axes of model and data, from which we derive the
much higher compute optimal data-model ratio than Chinchilla Optimal.
Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE
and MiniCPM-128K, whose excellent performance further cementing MiniCPM's
foundation in diverse SLM applications. MiniCPM models are available publicly
at https://github.com/OpenBMB/MiniCPM .
OmniFusion Technical Report
Authors: name: Elizaveta Goncharova, name: Anton Razzhigaev, name: Matvey Mikhalchuk, name: Maxim Kurkin, name: Irina Abdullaeva, name: Matvey Skripkin, name: Ivan Oseledets, name: Denis Dimitrov, name: Andrey Kuznetsov
Published: 2024-04-09 11:00:19

Last year, multimodal architectures served up a revolution in AI-based
approaches and solutions, extending the capabilities of large language models
(LLM). We propose an OmniFusion model based on a pretrained LLM and
adapters for visual modality. We evaluated and compared several architecture
design principles for better text and visual data coupling: MLP and transformer
adapters, various CLIP ViT-based encoders (SigLIP, InternVIT, etc.), and their
fusing approach, image encoding method (whole image or tiles encoding) and two
7B LLMs (the proprietary one and open-source Mistral). Experiments on 8
visual-language benchmarks show the top score for the best OmniFusion setup in
terms of different VQA tasks in comparison with open-source LLaVA-like
solutions: VizWiz, Pope, MM-Vet, ScienceQA, MMBench, TextVQA, VQAv2, MMMU. We
also propose a variety of situations, where OmniFusion provides highly-detailed
answers in different domains: housekeeping, sightseeing, culture, medicine,
handwritten and scanned equations recognition, etc. Mistral-based OmniFusion
model is an open-source solution with weights, training and inference scripts
available at https://github.com/AIRI-Institute/OmniFusion.
Elephants Never Forget: Memorization and Learning of Tabular Data in
  Large Language Models
Authors: name: Sebastian Bordt, name: Harsha Nori, name: Vanessa Rodrigues, name: Besmira Nushi, name: Rich Caruana
Published: 2024-04-09 10:58:21

While many have shown how Large Language Models (LLMs) can be applied to a
diverse set of tasks, the critical issues of data contamination and
memorization are often glossed over. In this work, we address this concern for
tabular data. Specifically, we introduce a variety of different techniques to
assess whether a language model has seen a tabular dataset during training.
This investigation reveals that LLMs have memorized many popular tabular
datasets verbatim. We then compare the few-shot learning performance of LLMs on
datasets that were seen during training to the performance on datasets released
after training. We find that LLMs perform better on datasets seen during
training, indicating that memorization leads to overfitting. At the same time,
LLMs show non-trivial performance on novel datasets and are surprisingly robust
to data transformations. We then investigate the in-context statistical
learning abilities of LLMs. Without fine-tuning, we find them to be limited.
This suggests that much of the few-shot performance on novel datasets is due to
the LLM's world knowledge. Overall, our results highlight the importance of
testing whether an LLM has seen an evaluation dataset during pre-training. We
make the exposure tests we developed available as the tabmemcheck Python
package at https://github.com/interpretml/LLM-Tabular-Memorization-Checker
CodecLM: Aligning Language Models with Tailored Synthetic Data
Authors: name: Zifeng Wang, name: Chun-Liang Li, name: Vincent Perot, name: Long T. Le, name: Jin Miao, name: Zizhao Zhang, name: Chen-Yu Lee, name: Tomas Pfister
Published: 2024-04-08 21:15:36

Instruction tuning has emerged as the key in aligning large language models
(LLMs) with specific task instructions, thereby mitigating the discrepancy
between the next-token prediction objective and users' actual goals. To reduce
the labor and time cost to collect or annotate data by humans, researchers
start to explore the use of LLMs to generate instruction-aligned synthetic
data. Recent works focus on generating diverse instructions and applying LLM to
increase instruction complexity, often neglecting downstream use cases. It
remains unclear how to tailor high-quality data to elicit better
instruction-following abilities in different target instruction distributions
and LLMs. To this end, we introduce CodecLM, a general framework for adaptively
generating high-quality synthetic data for LLM alignment with different
downstream instruction distributions and LLMs. Drawing on the Encode-Decode
principles, we use LLMs as codecs to guide the data generation process. We
first encode seed instructions into metadata, which are concise keywords
generated on-the-fly to capture the target instruction distribution, and then
decode metadata to create tailored instructions. We also introduce Self-Rubrics
and Contrastive Filtering during decoding to tailor data-efficient samples.
Extensive experiments on four open-domain instruction following benchmarks
validate the effectiveness of CodecLM over the current state-of-the-arts.