Orionnnn/hub-stats huggingface dataset:
Aurora-M: The First Open Source Multilingual Language Model Red-teamed
  according to the U.S. Executive Order
Authors: name: Taishi Nakamura, name: Mayank Mishra, name: Simone Tedeschi, name: Yekun Chai, name: Jason T Stillerman, name: Felix Friedrich, name: Prateek Yadav, name: Tanmay Laud, name: Vu Minh Chien, name: Terry Yue Zhuo, name: Diganta Misra, name: Ben Bogin, name: Xuan-Son Vu, name: Marzena Karpinska, name: Arnav Varma Dantuluri, name: Wojciech Kusa, name: Tommaso Furlanello, name: Rio Yokota, name: Niklas Muennighoff, name: Suhas Pai, name: Tosin Adewumi, name: Veronika Laippala, name: Xiaozhe Yao, name: Adalberto Junior, name: Alpay Ariyak, name: Aleksandr Drozd, name: Jordan Clive, name: Kshitij Gupta, name: Liangyu Chen, name: Qi Sun, name: Ken Tsui, name: Noah Persaud, name: Nour Fahmy, name: Tianlong Chen, name: Mohit Bansal, name: Nicolo Monti, name: Tai Dang, name: Ziyang Luo, name: Tien-Tung Bui, name: Roberto Navigli, name: Virendra Mehta, name: Matthew Blumberg, name: Victor May, name: Huu Nguyen, name: Sampo Pyysalo
Published: 2024-03-30 15:38:54

Pretrained language models underpin several AI applications, but their high
computational cost for training limits accessibility. Initiatives such as BLOOM
and StarCoder aim to democratize access to pretrained models for collaborative
community development. However, such existing models face challenges: limited
multilingual capabilities, continual pretraining causing catastrophic
forgetting, whereas pretraining from scratch is computationally expensive, and
compliance with AI safety and development laws. This paper presents Aurora-M, a
15B parameter multilingual open-source model trained on English, Finnish,
Hindi, Japanese, Vietnamese, and code. Continually pretrained from
StarCoderPlus on 435 billion additional tokens, Aurora-M surpasses 2 trillion
tokens in total training token count. It is the first open-source multilingual
model fine-tuned on human-reviewed safety instructions, thus aligning its
development not only with conventional red-teaming considerations, but also
with the specific concerns articulated in the Biden-Harris Executive Order on
the Safe, Secure, and Trustworthy Development and Use of Artificial
Intelligence. Aurora-M is rigorously evaluated across various tasks and
languages, demonstrating robustness against catastrophic forgetting and
outperforming alternatives in multilingual settings, particularly in safety
evaluations. To promote responsible open-source LLM development, Aurora-M and
its variants are released at
https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407 .
Measuring Style Similarity in Diffusion Models
Authors: name: Gowthami Somepalli, name: Anubhav Gupta, name: Kamal Gupta, name: Shramay Palta, name: Micah Goldblum, name: Jonas Geiping, name: Abhinav Shrivastava, name: Tom Goldstein
Published: 2024-04-01 17:58:30

Generative models are now widely used by graphic designers and artists. Prior
works have shown that these models remember and often replicate content from
their training data during generation. Hence as their proliferation increases,
it has become important to perform a database search to determine whether the
properties of the image are attributable to specific training data, every time
before a generated image is used for professional purposes. Existing tools for
this purpose focus on retrieving images of similar semantic content. Meanwhile,
many artists are concerned with style replication in text-to-image models. We
present a framework for understanding and extracting style descriptors from
images. Our framework comprises a new dataset curated using the insight that
style is a subjective property of an image that captures complex yet meaningful
interactions of factors including but not limited to colors, textures, shapes,
etc. We also propose a method to extract style descriptors that can be used to
attribute style of a generated image to the images used in the training dataset
of a text-to-image model. We showcase promising results in various style
retrieval tasks. We also quantitatively and qualitatively analyze style
attribution and matching in the Stable Diffusion model. Code and artifacts are
available at https://github.com/learn2phoenix/CSD.
ST-LLM: Large Language Models Are Effective Temporal Learners
Authors: name: Ruyang Liu, name: Chen Li, name: Haoran Tang, name: Yixiao Ge, name: Ying Shan, name: Ge Li
Published: 2024-03-30 10:11:26

Large Language Models (LLMs) have showcased impressive capabilities in text
comprehension and generation, prompting research efforts towards video LLMs to
facilitate human-AI interaction at the video level. However, how to effectively
encode and understand videos in video-based dialogue systems remains to be
solved. In this paper, we investigate a straightforward yet unexplored
question: Can we feed all spatial-temporal tokens into the LLM, thus delegating
the task of video sequence modeling to the LLMs? Surprisingly, this simple
approach yields significant improvements in video understanding. Based upon
this, we propose ST-LLM, an effective video-LLM baseline with Spatial-Temporal
sequence modeling inside LLM. Furthermore, to address the overhead and
stability issues introduced by uncompressed video tokens within LLMs, we
develop a dynamic masking strategy with tailor-made training objectives. For
particularly long videos, we have also designed a global-local input module to
balance efficiency and effectiveness. Consequently, we harness LLM for
proficient spatial-temporal modeling, while upholding efficiency and stability.
Extensive experimental results attest to the effectiveness of our method.
Through a more concise model and training pipeline, ST-LLM establishes a new
state-of-the-art result on VideoChatGPT-Bench and MVBench. Codes have been
available at https://github.com/TencentARC/ST-LLM.
Streaming Dense Video Captioning
Authors: name: Xingyi Zhou, name: Anurag Arnab, name: Shyamal Buch, name: Shen Yan, name: Austin Myers, name: Xuehan Xiong, name: Arsha Nagrani, name: Cordelia Schmid
Published: 2024-04-01 17:59:15

An ideal model for dense video captioning -- predicting captions localized
temporally in a video -- should be able to handle long input videos, predict
rich, detailed textual descriptions, and be able to produce outputs before
processing the entire video. Current state-of-the-art models, however, process
a fixed number of downsampled frames, and make a single full prediction after
seeing the whole video. We propose a streaming dense video captioning model
that consists of two novel components: First, we propose a new memory module,
based on clustering incoming tokens, which can handle arbitrarily long videos
as the memory is of a fixed size. Second, we develop a streaming decoding
algorithm that enables our model to make predictions before the entire video
has been processed. Our model achieves this streaming ability, and
significantly improves the state-of-the-art on three dense video captioning
benchmarks: ActivityNet, YouCook2 and ViTT. Our code is released at
https://github.com/google-research/scenic.
CosmicMan: A Text-to-Image Foundation Model for Humans
Authors: name: Shikai Li, name: Jianglin Fu, name: Kaiyuan Liu, name: Wentao Wang, name: Kwan-Yee Lin, name: Wayne Wu
Published: 2024-04-01 17:59:05

We present CosmicMan, a text-to-image foundation model specialized for
generating high-fidelity human images. Unlike current general-purpose
foundation models that are stuck in the dilemma of inferior quality and
text-image misalignment for humans, CosmicMan enables generating
photo-realistic human images with meticulous appearance, reasonable structure,
and precise text-image alignment with detailed dense descriptions. At the heart
of CosmicMan's success are the new reflections and perspectives on data and
models: (1) We found that data quality and a scalable data production flow are
essential for the final results from trained models. Hence, we propose a new
data production paradigm, Annotate Anyone, which serves as a perpetual data
flywheel to produce high-quality data with accurate yet cost-effective
annotations over time. Based on this, we constructed a large-scale dataset,
CosmicMan-HQ 1.0, with 6 Million high-quality real-world human images in a mean
resolution of 1488x1255, and attached with precise text annotations deriving
from 115 Million attributes in diverse granularities. (2) We argue that a
text-to-image foundation model specialized for humans must be pragmatic -- easy
to integrate into down-streaming tasks while effective in producing
high-quality human images. Hence, we propose to model the relationship between
dense text descriptions and image pixels in a decomposed manner, and present
Decomposed-Attention-Refocusing (Daring) training framework. It seamlessly
decomposes the cross-attention features in existing text-to-image diffusion
model, and enforces attention refocusing without adding extra modules. Through
Daring, we show that explicitly discretizing continuous text space into several
basic groups that align with human body structure is the key to tackling the
misalignment problem in a breeze.
Getting it Right: Improving Spatial Consistency in Text-to-Image Models
Authors: name: Agneet Chatterjee, name: Gabriela Ben Melech Stan, name: Estelle Aflalo, name: Sayak Paul, name: Dhruba Ghosh, name: Tejas Gokhale, name: Ludwig Schmidt, name: Hannaneh Hajishirzi, name: Vasudev Lal, name: Chitta Baral, name: Yezhou Yang
Published: 2024-04-01 15:55:25

One of the key shortcomings in current text-to-image (T2I) models is their
inability to consistently generate images which faithfully follow the spatial
relationships specified in the text prompt. In this paper, we offer a
comprehensive investigation of this limitation, while also developing datasets
and methods that achieve state-of-the-art performance. First, we find that
current vision-language datasets do not represent spatial relationships well
enough; to alleviate this bottleneck, we create SPRIGHT, the first
spatially-focused, large scale dataset, by re-captioning 6 million images from
4 widely used vision datasets. Through a 3-fold evaluation and analysis
pipeline, we find that SPRIGHT largely improves upon existing datasets in
capturing spatial relationships. To demonstrate its efficacy, we leverage only
~0.25% of SPRIGHT and achieve a 22% improvement in generating spatially
accurate images while also improving the FID and CMMD scores. Secondly, we find
that training on images containing a large number of objects results in
substantial improvements in spatial consistency. Notably, we attain
state-of-the-art on T2I-CompBench with a spatial score of 0.2133, by
fine-tuning on <500 images. Finally, through a set of controlled experiments
and ablations, we document multiple findings that we believe will enhance the
understanding of factors that affect spatial consistency in text-to-image
models. We publicly release our dataset and model to foster further research in
this area.
FlexiDreamer: Single Image-to-3D Generation with FlexiCubes
Authors: name: Ruowen Zhao, name: Zhengyi Wang, name: Yikai Wang, name: Zihan Zhou, name: Jun Zhu
Published: 2024-04-01 08:20:18

3D content generation from text prompts or single images has made remarkable
progress in quality and speed recently. One of its dominant paradigms involves
generating consistent multi-view images followed by a sparse-view
reconstruction. However, due to the challenge of directly deforming the mesh
representation to approach the target topology, most methodologies learn an
implicit representation (such as NeRF) during the sparse-view reconstruction
and acquire the target mesh by a post-processing extraction. Although the
implicit representation can effectively model rich 3D information, its training
typically entails a long convergence time. In addition, the post-extraction
operation from the implicit field also leads to undesirable visual artifacts.
In this paper, we propose FlexiDreamer, a novel single image-to-3d generation
framework that reconstructs the target mesh in an end-to-end manner. By
leveraging a flexible gradient-based extraction known as FlexiCubes, our method
circumvents the defects brought by the post-processing and facilitates a direct
acquisition of the target mesh. Furthermore, we incorporate a multi-resolution
hash grid encoding scheme that progressively activates the encoding levels into
the implicit field in FlexiCubes to help capture geometric details for per-step
optimization. Notably, FlexiDreamer recovers a dense 3D structure from a
single-view image in approximately 1 minute on a single NVIDIA A100 GPU,
outperforming previous methodologies by a large margin.
Direct Preference Optimization of Video Large Multimodal Models from
  Language Model Reward
Authors: name: Ruohong Zhang, name: Liangke Gui, name: Zhiqing Sun, name: Yihao Feng, name: Keyang Xu, name: Yuanhan Zhang, name: Di Fu, name: Chunyuan Li, name: Alexander Hauptmann, name: Yonatan Bisk, name: Yiming Yang
Published: 2024-04-01 17:28:16

Preference modeling techniques, such as direct preference optimization (DPO),
has shown effective in enhancing the generalization abilities of large language
model (LLM). However, in tasks involving video instruction-following, providing
informative feedback, especially for detecting hallucinations in generated
responses, remains a significant challenge. Previous studies have explored
using large large multimodal models (LMMs) as reward models to guide preference
modeling, but their ability to accurately assess the factuality of generated
responses compared to corresponding videos has not been conclusively
established. This paper introduces a novel framework that utilizes detailed
video captions as a proxy of video content, enabling language models to
incorporate this information as supporting evidence for scoring video Question
Answering (QA) predictions. Our approach demonstrates robust alignment with
OpenAI GPT-4V model's reward mechanism, which directly takes video frames as
input. Furthermore, we show that applying this tailored reward through DPO
significantly improves the performance of video LMMs on video QA tasks.
Condition-Aware Neural Network for Controlled Image Generation
Authors: name: Han Cai, name: Muyang Li, name: Zhuoyang Zhang, name: Qinsheng Zhang, name: Ming-Yu Liu, name: Song Han
Published: 2024-04-01 14:42:57

We present Condition-Aware Neural Network (CAN), a new method for adding
control to image generative models. In parallel to prior conditional control
methods, CAN controls the image generation process by dynamically manipulating
the weight of the neural network. This is achieved by introducing a
condition-aware weight generation module that generates conditional weight for
convolution/linear layers based on the input condition. We test CAN on
class-conditional image generation on ImageNet and text-to-image generation on
COCO. CAN consistently delivers significant improvements for diffusion
transformer models, including DiT and UViT. In particular, CAN combined with
EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2
while requiring 52x fewer MACs per sampling step.
MaGRITTe: Manipulative and Generative 3D Realization from Image, Topview
  and Text
Authors: name: Takayuki Hara, name: Tatsuya Harada
Published: 2024-03-30 12:50:25

The generation of 3D scenes from user-specified conditions offers a promising
avenue for alleviating the production burden in 3D applications. Previous
studies required significant effort to realize the desired scene, owing to
limited control conditions. We propose a method for controlling and generating
3D scenes under multimodal conditions using partial images, layout information
represented in the top view, and text prompts. Combining these conditions to
generate a 3D scene involves the following significant difficulties: (1) the
creation of large datasets, (2) reflection on the interaction of multimodal
conditions, and (3) domain dependence of the layout conditions. We decompose
the process of 3D scene generation into 2D image generation from the given
conditions and 3D scene generation from 2D images. 2D image generation is
achieved by fine-tuning a pretrained text-to-image model with a small
artificial dataset of partial images and layouts, and 3D scene generation is
achieved by layout-conditioned depth estimation and neural radiance fields
(NeRF), thereby avoiding the creation of large datasets. The use of a common
representation of spatial information using 360-degree images allows for the
consideration of multimodal condition interactions and reduces the domain
dependence of the layout control. The experimental results qualitatively and
quantitatively demonstrated that the proposed method can generate 3D scenes in
diverse domains, from indoor to outdoor, according to multimodal conditions.
Localizing Paragraph Memorization in Language Models
Authors: name: Niklas Stoehr, name: Mitchell Gordon, name: Chiyuan Zhang, name: Owen Lewis
Published: 2024-03-28 21:53:24

Can we localize the weights and mechanisms used by a language model to
memorize and recite entire paragraphs of its training data? In this paper, we
show that while memorization is spread across multiple layers and model
components, gradients of memorized paragraphs have a distinguishable spatial
pattern, being larger in lower model layers than gradients of non-memorized
examples. Moreover, the memorized examples can be unlearned by fine-tuning only
the high-gradient weights. We localize a low-layer attention head that appears
to be especially involved in paragraph memorization. This head is predominantly
focusing its attention on distinctive, rare tokens that are least frequent in a
corpus-level unigram distribution. Next, we study how localized memorization is
across the tokens in the prefix by perturbing tokens and measuring the caused
change in the decoding. A few distinctive tokens early in a prefix can often
corrupt the entire continuation. Overall, memorized continuations are not only
harder to unlearn, but also to corrupt than non-memorized ones.
Transformer-Lite: High-efficiency Deployment of Large Language Models on
  Mobile Phone GPUs
Authors: name: Luchang Li, name: Sheng Qian, name: Jie Lu, name: Lunxi Yuan, name: Rui Wang, name: Qin Xie
Published: 2024-03-29 08:26:53

The Large Language Model (LLM) is widely employed for tasks such as
intelligent assistants, text summarization, translation, and multi-modality on
mobile phones. However, the current methods for on-device LLM deployment
maintain slow inference speed, which causes poor user experience. To facilitate
high-efficiency LLM deployment on device GPUs, we propose four optimization
techniques: (a) a symbolic expression-based approach to support dynamic shape
model inference; (b) operator optimizations and execution priority setting to
enhance inference speed and reduce phone lagging; (c) an FP4 quantization
method termed M0E4 to reduce dequantization overhead; (d) a sub-tensor-based
technique to eliminate the need for copying KV cache after LLM inference.
Furthermore, we implement these methods in our mobile inference engine,
Transformer-Lite, which is compatible with both Qualcomm and MTK processors. We
evaluated Transformer-Lite's performance using LLMs with varied architectures
and parameters ranging from 2B to 14B. Specifically, we achieved prefill and
decoding speeds of 121 token/s and 14 token/s for ChatGLM2 6B, and 330 token/s
and 30 token/s for smaller Gemma 2B, respectively. Compared with CPU-based
FastLLM and GPU-based MLC-LLM, our engine attains over 10x speedup for the
prefill speed and 2~3x speedup for the decoding speed.
MambaMixer: Efficient Selective State Space Models with Dual Token and
  Channel Selection
Authors: name: Ali Behrouz, name: Michele Santacatterina, name: Ramin Zabih
Published: 2024-03-29 00:05:13

Recent advances in deep learning have mainly relied on Transformers due to
their data dependency and ability to learn at scale. The attention module in
these architectures, however, exhibits quadratic time and space in input size,
limiting their scalability for long-sequence modeling. Despite recent attempts
to design efficient and effective architecture backbone for multi-dimensional
data, such as images and multivariate time series, existing models are either
data independent, or fail to allow inter- and intra-dimension communication.
Recently, State Space Models (SSMs), and more specifically Selective State
Space Models, with efficient hardware-aware implementation, have shown
promising potential for long sequence modeling. Motivated by the success of
SSMs, we present MambaMixer, a new architecture with data-dependent weights
that uses a dual selection mechanism across tokens and channels, called
Selective Token and Channel Mixer. MambaMixer connects selective mixers using a
weighted averaging mechanism, allowing layers to have direct access to early
features. As a proof of concept, we design Vision MambaMixer (ViM2) and Time
Series MambaMixer (TSM2) architectures based on the MambaMixer block and
explore their performance in various vision and time series forecasting tasks.
Our results underline the importance of selective mixing across both tokens and
channels. In ImageNet classification, object detection, and semantic
segmentation tasks, ViM2 achieves competitive performance with well-established
vision models and outperforms SSM-based vision models. In time series
forecasting, TSM2 achieves outstanding performance compared to state-of-the-art
methods while demonstrating significantly improved computational cost. These
results show that while Transformers, cross-channel attention, and MLPs are
sufficient for good performance in time series forecasting, neither is
necessary.
DiJiang: Efficient Large Language Models through Compact Kernelization
Authors: name: Hanting Chen, name: Zhicheng Liu, name: Xutao Wang, name: Yuchuan Tian, name: Yunhe Wang
Published: 2024-03-29 02:32:15

In an effort to reduce the computational load of Transformers, research on
linear attention has gained significant momentum. However, the improvement
strategies for attention mechanisms typically necessitate extensive retraining,
which is impractical for large language models with a vast array of parameters.
In this paper, we present DiJiang, a novel Frequency Domain Kernelization
approach that enables the transformation of a pre-trained vanilla Transformer
into a linear complexity model with little training costs. By employing a
weighted Quasi-Monte Carlo method for sampling, the proposed approach
theoretically offers superior approximation efficiency. To further reduce the
training computational complexity, our kernelization is based on Discrete
Cosine Transform (DCT) operations. Extensive experiments demonstrate that the
proposed method achieves comparable performance to the original Transformer,
but with significantly reduced training costs and much faster inference speeds.
Our DiJiang-7B achieves comparable performance with LLaMA2-7B on various
benchmark while requires only about 1/50 training cost. Code is available at
https://github.com/YuchuanTian/DiJiang.