Orionnnn/hub-stats huggingface dataset:
On the Scalability of Diffusion-based Text-to-Image Generation
Authors: name: Hao Li, name: Yang Zou, name: Ying Wang, name: Orchid Majumder, name: Yusheng Xie, name: R. Manmatha, name: Ashwin Swaminathan, name: Zhuowen Tu, name: Stefano Ermon, name: Stefano Soatto
Published: 2024-04-03 17:34:28

Scaling up model and data size has been quite successful for the evolution of
LLMs. However, the scaling law for the diffusion based text-to-image (T2I)
models is not fully explored. It is also unclear how to efficiently scale the
model for better performance at reduced cost. The different training settings
and expensive training cost make a fair model comparison extremely difficult.
In this work, we empirically study the scaling properties of diffusion based
T2I models by performing extensive and rigours ablations on scaling both
denoising backbones and training set, including training scaled UNet and
Transformer variants ranging from 0.4B to 4B parameters on datasets upto 600M
images. For model scaling, we find the location and amount of cross attention
distinguishes the performance of existing UNet designs. And increasing the
transformer blocks is more parameter-efficient for improving text-image
alignment than increasing channel numbers. We then identify an efficient UNet
variant, which is 45% smaller and 28% faster than SDXL's UNet. On the data
scaling side, we show the quality and diversity of the training set matters
more than simply dataset size. Increasing caption density and diversity
improves text-image alignment performance and the learning efficiency. Finally,
we provide scaling functions to predict the text-image alignment performance as
functions of the scale of model size, compute and dataset size.
InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image
  Generation
Authors: name: Haofan Wang, name: Qixun Wang, name: Xu Bai, name: Zekui Qin, name: Anthony Chen
Published: 2024-04-03 13:34:09

Tuning-free diffusion-based models have demonstrated significant potential in
the realm of image personalization and customization. However, despite this
notable progress, current models continue to grapple with several complex
challenges in producing style-consistent image generation. Firstly, the concept
of style is inherently underdetermined, encompassing a multitude of elements
such as color, material, atmosphere, design, and structure, among others.
Secondly, inversion-based methods are prone to style degradation, often
resulting in the loss of fine-grained details. Lastly, adapter-based approaches
frequently require meticulous weight tuning for each reference image to achieve
a balance between style intensity and text controllability. In this paper, we
commence by examining several compelling yet frequently overlooked
observations. We then proceed to introduce InstantStyle, a framework designed
to address these issues through the implementation of two key strategies: 1) A
straightforward mechanism that decouples style and content from reference
images within the feature space, predicated on the assumption that features
within the same space can be either added to or subtracted from one another. 2)
The injection of reference image features exclusively into style-specific
blocks, thereby preventing style leaks and eschewing the need for cumbersome
weight tuning, which often characterizes more parameter-heavy designs.Our work
demonstrates superior visual stylization outcomes, striking an optimal balance
between the intensity of style and the controllability of textual elements. Our
codes will be available at https://github.com/InstantStyle/InstantStyle.
3D Congealing: 3D-Aware Image Alignment in the Wild
Authors: name: Yunzhi Zhang, name: Zizhang Li, name: Amit Raj, name: Andreas Engelhardt, name: Yuanzhen Li, name: Tingbo Hou, name: Jiajun Wu, name: Varun Jampani
Published: 2024-04-02 17:32:12

We propose 3D Congealing, a novel problem of 3D-aware alignment for 2D images
capturing semantically similar objects. Given a collection of unlabeled
Internet images, our goal is to associate the shared semantic parts from the
inputs and aggregate the knowledge from 2D images to a shared 3D canonical
space. We introduce a general framework that tackles the task without assuming
shape templates, poses, or any camera parameters. At its core is a canonical 3D
representation that encapsulates geometric and semantic information. The
framework optimizes for the canonical representation together with the pose for
each input image, and a per-image coordinate map that warps 2D pixel
coordinates to the 3D canonical frame to account for the shape matching. The
optimization procedure fuses prior knowledge from a pre-trained image
generative model and semantic information from input images. The former
provides strong knowledge guidance for this under-constraint task, while the
latter provides the necessary information to mitigate the training data bias
from the pre-trained model. Our framework can be used for various tasks such as
correspondence matching, pose estimation, and image editing, achieving strong
results on real-world image datasets under challenging illumination conditions
and on in-the-wild online image collections.
Octopus v2: On-device language model for super agent
Authors: name: Wei Chen, name: Zhiyuan Li
Published: 2024-04-02 09:01:32

Language models have shown effectiveness in a variety of software
applications, particularly in tasks related to automatic workflow. These models
possess the crucial ability to call functions, which is essential in creating
AI agents. Despite the high performance of large-scale language models in cloud
environments, they are often associated with concerns over privacy and cost.
Current on-device models for function calling face issues with latency and
accuracy. Our research presents a new method that empowers an on-device model
with 2 billion parameters to surpass the performance of GPT-4 in both accuracy
and latency, and decrease the context length by 95\%. When compared to Llama-7B
with a RAG-based function calling mechanism, our method enhances latency by
35-fold. This method reduces the latency to levels deemed suitable for
deployment across a variety of edge devices in production environments,
aligning with the performance requisites for real-world applications.
Poro 34B and the Blessing of Multilinguality
Authors: name: Risto Luukkonen, name: Jonathan Burdge, name: Elaine Zosa, name: Aarne Talman, name: Ville Komulainen, name: Väinö Hatanpää, name: Peter Sarlin, name: Sampo Pyysalo
Published: 2024-04-02 11:34:12

The pretraining of state-of-the-art large language models now requires
trillions of words of text, which is orders of magnitude more than available
for the vast majority of languages. While including text in more than one
language is an obvious way to acquire more pretraining data, multilinguality is
often seen as a curse, and most model training efforts continue to focus
near-exclusively on individual large languages. We believe that multilinguality
can be a blessing and that it should be possible to substantially improve over
the capabilities of monolingual models for small languages through multilingual
training. In this study, we introduce Poro 34B, a 34 billion parameter model
trained for 1 trillion tokens of Finnish, English, and programming languages,
and demonstrate that a multilingual training approach can produce a model that
not only substantially advances over the capabilities of existing models for
Finnish, but also excels in translation and is competitive in its class in
generating English and programming languages. We release the model parameters,
scripts, and data under open licenses at
https://huggingface.co/LumiOpen/Poro-34B.
Bigger is not Always Better: Scaling Properties of Latent Diffusion
  Models
Authors: name: Kangfu Mei, name: Zhengzhong Tu, name: Mauricio Delbracio, name: Hossein Talebi, name: Vishal M. Patel, name: Peyman Milanfar
Published: 2024-04-01 17:59:48

We study the scaling properties of latent diffusion models (LDMs) with an
emphasis on their sampling efficiency. While improved network architecture and
inference algorithms have shown to effectively boost sampling efficiency of
diffusion models, the role of model size -- a critical determinant of sampling
efficiency -- has not been thoroughly examined. Through empirical analysis of
established text-to-image diffusion models, we conduct an in-depth
investigation into how model size influences sampling efficiency across varying
sampling steps. Our findings unveil a surprising trend: when operating under a
given inference budget, smaller models frequently outperform their larger
equivalents in generating high-quality results. Moreover, we extend our study
to demonstrate the generalizability of the these findings by applying various
diffusion samplers, exploring diverse downstream tasks, evaluating
post-distilled models, as well as comparing performance relative to training
compute. These findings open up new pathways for the development of LDM scaling
strategies which can be employed to enhance generative capabilities within
limited inference budgets.
LLM-ABR: Designing Adaptive Bitrate Algorithms via Large Language Models
Authors: name: Zhiyuan He, name: Aashish Gottipati, name: Lili Qiu, name: Francis Y. Yan, name: Xufang Luo, name: Kenuo Xu, name: Yuqing Yang
Published: 2024-04-02 03:43:55

We present LLM-ABR, the first system that utilizes the generative
capabilities of large language models (LLMs) to autonomously design adaptive
bitrate (ABR) algorithms tailored for diverse network characteristics.
Operating within a reinforcement learning framework, LLM-ABR empowers LLMs to
design key components such as states and neural network architectures. We
evaluate LLM-ABR across diverse network settings, including broadband,
satellite, 4G, and 5G. LLM-ABR consistently outperforms default ABR algorithms.
Are large language models superhuman chemists?
Authors: name: Adrian Mirza, name: Nawaf Alampara, name: Sreekanth Kunchapu, name: Benedict Emoekabu, name: Aswanth Krishnan, name: Mara Wilhelmi, name: Macjonathan Okereke, name: Juliane Eberhardt, name: Amir Mohammad Elahi, name: Maximilian Greiner, name: Caroline T. Holick, name: Tanya Gupta, name: Mehrdad Asgari, name: Christina Glaubitz, name: Lea C. Klepsch, name: Yannik Köster, name: Jakob Meyer, name: Santiago Miret, name: Tim Hoffmann, name: Fabian Alexander Kreth, name: Michael Ringleb, name: Nicole Roesner, name: Ulrich S. Schubert, name: Leanne M. Stafast, name: Dinga Wonanke, name: Michael Pieler, name: Philippe Schwaller, name: Kevin Maik Jablonka
Published: 2024-04-01 20:56:25

Large language models (LLMs) have gained widespread interest due to their
ability to process human language and perform tasks on which they have not been
explicitly trained. This is relevant for the chemical sciences, which face the
problem of small and diverse datasets that are frequently in the form of text.
LLMs have shown promise in addressing these issues and are increasingly being
harnessed to predict chemical properties, optimize reactions, and even design
and conduct experiments autonomously. However, we still have only a very
limited systematic understanding of the chemical reasoning capabilities of
LLMs, which would be required to improve models and mitigate potential harms.
Here, we introduce "ChemBench," an automated framework designed to rigorously
evaluate the chemical knowledge and reasoning abilities of state-of-the-art
LLMs against the expertise of human chemists. We curated more than 7,000
question-answer pairs for a wide array of subfields of the chemical sciences,
evaluated leading open and closed-source LLMs, and found that the best models
outperformed the best human chemists in our study on average. The models,
however, struggle with some chemical reasoning tasks that are easy for human
experts and provide overconfident, misleading predictions, such as about
chemicals' safety profiles. These findings underscore the dual reality that,
although LLMs demonstrate remarkable proficiency in chemical tasks, further
research is critical to enhancing their safety and utility in chemical
sciences. Our findings also indicate a need for adaptations to chemistry
curricula and highlight the importance of continuing to develop evaluation
frameworks to improve safe and useful LLMs.
Advancing LLM Reasoning Generalists with Preference Trees
Authors: name: Lifan Yuan, name: Ganqu Cui, name: Hanbin Wang, name: Ning Ding, name: Xingyao Wang, name: Jia Deng, name: Boji Shan, name: Huimin Chen, name: Ruobing Xie, name: Yankai Lin, name: Zhenghao Liu, name: Bowen Zhou, name: Hao Peng, name: Zhiyuan Liu, name: Maosong Sun
Published: 2024-04-02 16:25:30

We introduce Eurus, a suite of large language models (LLMs) optimized for
reasoning. Finetuned from Mistral-7B and CodeLlama-70B, Eurus models achieve
state-of-the-art results among open-source models on a diverse set of
benchmarks covering mathematics, code generation, and logical reasoning
problems. Notably, Eurus-70B beats GPT-3.5 Turbo in reasoning through a
comprehensive benchmarking across 12 tests covering five tasks, and achieves a
33.3% pass@1 accuracy on LeetCode and 32.6% on TheoremQA, two challenging
benchmarks, substantially outperforming existing open-source models by margins
more than 13.3%. The strong performance of Eurus can be primarily attributed to
UltraInteract, our newly-curated large-scale, high-quality alignment dataset
specifically designed for complex reasoning tasks. UltraInteract can be used in
both supervised fine-tuning and preference learning. For each instruction, it
includes a preference tree consisting of (1) reasoning chains with diverse
planning strategies in a unified format, (2) multi-turn interaction
trajectories with the environment and the critique, and (3) pairwise data to
facilitate preference learning. UltraInteract allows us to conduct an in-depth
exploration of preference learning for reasoning tasks. Our investigation
reveals that some well-established preference learning algorithms may be less
suitable for reasoning tasks compared to their effectiveness in general
conversations. Inspired by this, we derive a novel reward modeling objective
which, together with UltraInteract, leads to a strong reward model.
LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
  Language Model
Authors: name: Musashi Hinck, name: Matthew L. Olson, name: David Cobbley, name: Shao-Yen Tseng, name: Vasudev Lal
Published: 2024-03-29 21:32:50

We train a suite of multimodal foundation models (MMFM) using the popular
LLaVA framework with the recently released Gemma family of large language
models (LLMs). Of particular interest is the 2B parameter Gemma model, which
provides opportunities to construct capable small-scale MMFMs. In line with
findings from other papers in this space, we test the effect of ablating three
design features: pretraining the connector, utilizing a more powerful image
backbone, and increasing the size of the language backbone. The resulting
models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
evaluations, but fail to improve past the current comparably sized SOTA models.
Closer analysis of performance shows mixed effects; skipping pretraining tends
to reduce performance, larger vision models sometimes improve performance, and
increasing language model size has inconsistent effects. We publicly release
training recipes, code and weights for our models for the LLaVA-Gemma models.
Long-context LLMs Struggle with Long In-context Learning
Authors: name: Tianle Li, name: Ge Zhang, name: Quy Duc Do, name: Xiang Yue, name: Wenhu Chen
Published: 2024-04-02 15:59:11

Large Language Models (LLMs) have made significant strides in handling long
sequences exceeding 32K tokens. However, their performance evaluation has
largely been confined to metrics like perplexity and synthetic tasks, which may
not fully capture their abilities in more nuanced, real-world scenarios. This
study introduces a specialized benchmark (LIConBench) focusing on long
in-context learning within the realm of extreme-label classification. We
meticulously selected six datasets with a label range spanning 28 to 174
classes covering different input (few-shot demonstration) length from 2K to
50K. Our benchmark requires LLMs to comprehend the entire input to recognize
the massive label spaces to make correct prediction. We evaluate 13
long-context LLMs on our benchmarks. We find that the long-context LLMs perform
relatively well under the token length of 20K and the performance benefits from
utilizing the long context window. However, after the context window exceeds
20K, most LLMs except GPT-4 will dip dramatically. This suggests a notable gap
in current LLM capabilities for processing and understanding long, context-rich
sequences. Further analysis revealed a tendency among models to favor
predictions for labels presented towards the end at the sequence. Their ability
to reason over multiple pieces in the long sequence is yet to be improved. Our
study reveals that long context understanding and reasoning is still a
challenging task for the existing LLMs. We believe LIConBench could serve as a
more realistic evaluation for the future long context LLMs.
CameraCtrl: Enabling Camera Control for Text-to-Video Generation
Authors: name: Hao He, name: Yinghao Xu, name: Yuwei Guo, name: Gordon Wetzstein, name: Bo Dai, name: Hongsheng Li, name: Ceyuan Yang
Published: 2024-04-02 16:52:41

Controllability plays a crucial role in video generation since it allows
users to create desired content. However, existing models largely overlooked
the precise control of camera pose that serves as a cinematic language to
express deeper narrative nuances. To alleviate this issue, we introduce
CameraCtrl, enabling accurate camera pose control for text-to-video(T2V)
models. After precisely parameterizing the camera trajectory, a plug-and-play
camera module is then trained on a T2V model, leaving others untouched.
Additionally, a comprehensive study on the effect of various datasets is also
conducted, suggesting that videos with diverse camera distribution and similar
appearances indeed enhance controllability and generalization. Experimental
results demonstrate the effectiveness of CameraCtrl in achieving precise and
domain-adaptive camera control, marking a step forward in the pursuit of
dynamic and customized video storytelling from textual and camera pose inputs.
Our project website is at: https://hehao13.github.io/projects-CameraCtrl/.
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Authors: name: Shujie Hu, name: Long Zhou, name: Shujie Liu, name: Sanyuan Chen, name: Hongkun Hao, name: Jing Pan, name: Xunying Liu, name: Jinyu Li, name: Sunit Sivasankaran, name: Linquan Liu, name: Furu Wei
Published: 2024-03-31 12:01:32

The recent advancements in large language models (LLMs) have revolutionized
the field of natural language processing, progressively broadening their scope
to multimodal perception and generation. However, effectively integrating
listening capabilities into LLMs poses significant challenges, particularly
with respect to generalizing across varied contexts and executing complex
auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech
large language model with dual encoders, and a prompt-aware LoRA weight
adapter, optimized by a two-stage curriculum learning approach. Leveraging dual
encoders, we decouple different types of speech information, utilizing a
Whisper encoder to process the semantic content of speech, and a WavLM encoder
to capture the unique characteristics of the speaker's identity. Within the
curriculum learning framework, WavLLM first builds its foundational
capabilities by optimizing on mixed elementary single tasks, followed by
advanced multi-task training on more complex tasks such as combinations of the
elementary tasks. To enhance the flexibility and adherence to different tasks
and instructions, a prompt-aware LoRA weight adapter is introduced in the
second advanced multi-task training stage. We validate the proposed model on
universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also
apply it to specialized datasets like Gaokao English listening comprehension
set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments
demonstrate that the proposed model achieves state-of-the-art performance
across a range of speech tasks on the same model size, exhibiting robust
generalization capabilities in executing complex tasks using CoT approach.
Furthermore, our model successfully completes Gaokao tasks without specialized
training. The codes, models, audio, and Gaokao evaluation set can be accessed
at aka.ms/wavllm.
Noise-Aware Training of Layout-Aware Language Models
Authors: name: Ritesh Sarkhel, name: Xiaoqi Ren, name: Lauro Beltrao Costa, name: Guolong Su, name: Vincent Perot, name: Yanan Xie, name: Emmanouil Koukoumidis, name: Arnab Nandi
Published: 2024-03-30 23:06:34

A visually rich document (VRD) utilizes visual features along with linguistic
cues to disseminate information. Training a custom extractor that identifies
named entities from a document requires a large number of instances of the
target document type annotated at textual and visual modalities. This is an
expensive bottleneck in enterprise scenarios, where we want to train custom
extractors for thousands of different document types in a scalable way.
Pre-training an extractor model on unlabeled instances of the target document
type, followed by a fine-tuning step on human-labeled instances does not work
in these scenarios, as it surpasses the maximum allowable training time
allocated for the extractor. We address this scenario by proposing a
Noise-Aware Training method or NAT in this paper. Instead of acquiring
expensive human-labeled documents, NAT utilizes weakly labeled documents to
train an extractor in a scalable way. To avoid degradation in the model's
quality due to noisy, weakly labeled samples, NAT estimates the confidence of
each training sample and incorporates it as uncertainty measure during
training. We train multiple state-of-the-art extractor models using NAT.
Experiments on a number of publicly available and in-house datasets show that
NAT-trained models are not only robust in performance -- it outperforms a
transfer-learning baseline by up to 6% in terms of macro-F1 score, but it is
also more label-efficient -- it reduces the amount of human-effort required to
obtain comparable performance by up to 73%.