Orionnnn/hub-stats huggingface dataset:
SEINE: Short-to-Long Video Diffusion Model for Generative Transition and
  Prediction
Authors: name: Xinyuan Chen, name: Yaohui Wang, name: Lingjun Zhang, name: Shaobin Zhuang, name: Xin Ma, name: Jiashuo Yu, name: Yali Wang, name: Dahua Lin, name: Yu Qiao, name: Ziwei Liu
Published: 2023-10-31 17:58:17

Recently video generation has achieved substantial progress with realistic
results. Nevertheless, existing AI-generated videos are usually very short
clips ("shot-level") depicting a single scene. To deliver a coherent long video
("story-level"), it is desirable to have creative transition and prediction
effects across different clips. This paper presents a short-to-long video
diffusion model, SEINE, that focuses on generative transition and prediction.
The goal is to generate high-quality long videos with smooth and creative
transitions between scenes and varying lengths of shot-level videos.
Specifically, we propose a random-mask video diffusion model to automatically
generate transitions based on textual descriptions. By providing the images of
different scenes as inputs, combined with text-based control, our model
generates transition videos that ensure coherence and visual quality.
Furthermore, the model can be readily extended to various tasks such as
image-to-video animation and autoregressive video prediction. To conduct a
comprehensive evaluation of this new generative task, we propose three
assessing criteria for smooth and creative transition: temporal consistency,
semantic similarity, and video-text semantic alignment. Extensive experiments
validate the effectiveness of our approach over existing methods for generative
transition and prediction, enabling the creation of story-level long videos.
Project page: https://vchitect.github.io/SEINE-project/ .
Beyond U: Making Diffusion Models Faster & Lighter
Authors: name: Sergio Calvo-Ordonez, name: Jiahao Huang, name: Lipei Zhang, name: Guang Yang, name: Carola-Bibiane Schonlieb, name: Angelica I Aviles-Rivero
Published: 2023-10-31 00:12:14

Diffusion models are a family of generative models that yield record-breaking
performance in tasks such as image synthesis, video generation, and molecule
design. Despite their capabilities, their efficiency, especially in the reverse
denoising process, remains a challenge due to slow convergence rates and high
computational costs. In this work, we introduce an approach that leverages
continuous dynamical systems to design a novel denoising network for diffusion
models that is more parameter-efficient, exhibits faster convergence, and
demonstrates increased noise robustness. Experimenting with denoising
probabilistic diffusion models, our framework operates with approximately a
quarter of the parameters and 30% of the Floating Point Operations (FLOPs)
compared to standard U-Nets in Denoising Diffusion Probabilistic Models
(DDPMs). Furthermore, our model is up to 70% faster in inference than the
baseline models when measured in equal conditions while converging to better
quality solutions.
What's In My Big Data?
Authors: name: Yanai Elazar, name: Akshita Bhagia, name: Ian Magnusson, name: Abhilasha Ravichander, name: Dustin Schwenk, name: Alane Suhr, name: Pete Walsh, name: Dirk Groeneveld, name: Luca Soldaini, name: Sameer Singh, name: Hanna Hajishirzi, name: Noah A. Smith, name: Jesse Dodge
Published: 2023-10-31 17:59:38

Large text corpora are the backbone of language models. However, we have a
limited understanding of the content of these corpora, including general
statistics, quality, social factors, and inclusion of evaluation data
(contamination). In this work, we propose What's In My Big Data? (WIMBD), a
platform and a set of sixteen analyses that allow us to reveal and compare the
contents of large text corpora. WIMBD builds on two basic capabilities -- count
and search -- at scale, which allows us to analyze more than 35 terabytes on a
standard compute node. We apply WIMBD to ten different corpora used to train
popular language models, including C4, The Pile, and RedPajama. Our analysis
uncovers several surprising and previously undocumented findings about these
corpora, including the high prevalence of duplicate, synthetic, and low-quality
content, personally identifiable information, toxic language, and benchmark
contamination. For instance, we find that about 50% of the documents in
RedPajama and LAION-2B-en are duplicates. In addition, several datasets used
for benchmarking models trained on such corpora are contaminated with respect
to important benchmarks, including the Winograd Schema Challenge and parts of
GLUE and SuperGLUE. We open-source WIMBD's code and artifacts to provide a
standard set of evaluations for new text-based corpora and to encourage more
analyses and transparency around them: github.com/allenai/wimbd.
LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
Authors: name: Simon Lermen, name: Charlie Rogers-Smith, name: Jeffrey Ladish
Published: 2023-10-31 16:55:06

AI developers often apply safety alignment procedures to prevent the misuse
of their AI systems. For example, before Meta released Llama 2-Chat, a
collection of instruction fine-tuned large language models, they invested
heavily in safety training, incorporating extensive red-teaming and
reinforcement learning from human feedback. However, it remains unclear how
well safety training guards against model misuse when attackers have access to
model weights. We explore the robustness of safety training in language models
by subversively fine-tuning the public weights of Llama 2-Chat. We employ
low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of
less than $200 per model and using only one GPU, we successfully undo the
safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B. Specifically,
our fine-tuning technique significantly reduces the rate at which the model
refuses to follow harmful instructions. We achieve a refusal rate below 1% for
our 70B Llama 2-Chat model on two refusal benchmarks. Our fine-tuning method
retains general performance, which we validate by comparing our fine-tuned
models against Llama 2-Chat across two benchmarks. Additionally, we present a
selection of harmful outputs produced by our models. While there is
considerable uncertainty about the scope of risks from current models, it is
likely that future models will have significantly more dangerous capabilities,
including the ability to hack into critical infrastructure, create dangerous
bio-weapons, or autonomously replicate and adapt to new environments. We show
that subversive fine-tuning is practical and effective, and hence argue that
evaluating risks from fine-tuning should be a core part of risk assessments for
releasing model weights.
CapsFusion: Rethinking Image-Text Data at Scale
Authors: name: Qiying Yu, name: Quan Sun, name: Xiaosong Zhang, name: Yufeng Cui, name: Fan Zhang, name: Xinlong Wang, name: Jingjing Liu
Published: 2023-10-31 15:31:39

Large multimodal models demonstrate remarkable generalist ability to perform
diverse multimodal tasks in a zero-shot manner. Large-scale web-based
image-text pairs contribute fundamentally to this success, but suffer from
excessive noise. Recent studies use alternative captions synthesized by
captioning models and have achieved notable benchmark performance. However, our
experiments reveal significant Scalability Deficiency and World Knowledge Loss
issues in models trained with synthetic captions, which have been largely
obscured by their initial benchmark success. Upon closer examination, we
identify the root cause as the overly-simplified language structure and lack of
knowledge details in existing synthetic captions. To provide higher-quality and
more scalable multimodal pretraining data, we propose CapsFusion, an advanced
framework that leverages large language models to consolidate and refine
information from both web-based image-text pairs and synthetic captions.
Extensive experiments show that CapsFusion captions exhibit remarkable
all-round superiority over existing captions in terms of model performance
(e.g., 18.8 and 18.3 improvements in CIDEr score on COCO and NoCaps), sample
efficiency (requiring 11-16 times less computation than baselines), world
knowledge depth, and scalability. These effectiveness, efficiency and
scalability advantages position CapsFusion as a promising candidate for future
scaling of LMM training.
Leveraging Word Guessing Games to Assess the Intelligence of Large
  Language Models
Authors: name: Tian Liang, name: Zhiwei He, name: Jen-tes Huang, name: Wenxuan Wang, name: Wenxiang Jiao, name: Rui Wang, name: Yujiu Yang, name: Zhaopeng Tu, name: Shuming Shi, name: Xing Wang
Published: 2023-10-31 14:37:42

The automatic evaluation of LLM-based agent intelligence is critical in
developing advanced LLM-based agents. Although considerable effort has been
devoted to developing human-annotated evaluation datasets, such as AlpacaEval,
existing techniques are costly, time-consuming, and lack adaptability. In this
paper, inspired by the popular language game ``Who is Spy'', we propose to use
the word guessing game to assess the intelligence performance of LLMs. Given a
word, the LLM is asked to describe the word and determine its identity (spy or
not) based on its and other players' descriptions. Ideally, an advanced agent
should possess the ability to accurately describe a given word using an
aggressive description while concurrently maximizing confusion in the
conservative description, enhancing its participation in the game. To this end,
we first develop DEEP to evaluate LLMs' expression and disguising abilities.
DEEP requires LLM to describe a word in aggressive and conservative modes. We
then introduce SpyGame, an interactive multi-agent framework designed to assess
LLMs' intelligence through participation in a competitive language-based board
game. Incorporating multi-agent interaction, SpyGame requires the target LLM to
possess linguistic skills and strategic thinking, providing a more
comprehensive evaluation of LLMs' human-like cognitive abilities and
adaptability in complex communication situations. The proposed evaluation
framework is very easy to implement. We collected words from multiple sources,
domains, and languages and used the proposed evaluation framework to conduct
experiments. Extensive experiments demonstrate that the proposed DEEP and
SpyGame effectively evaluate the capabilities of various LLMs, capturing their
ability to adapt to novel situations and engage in strategic communication.
The Impact of Depth and Width on Transformer Language Model
  Generalization
Authors: name: Jackson Petty, name: Sjoerd van Steenkiste, name: Ishita Dasgupta, name: Fei Sha, name: Dan Garrette, name: Tal Linzen
Published: 2023-10-30 19:10:06

To process novel sentences, language models (LMs) must generalize
compositionally -- combine familiar elements in new ways. What aspects of a
model's structure promote compositional generalization? Focusing on
transformers, we test the hypothesis, motivated by recent theoretical and
empirical work, that transformers generalize more compositionally when they are
deeper (have more layers). Because simply adding layers increases the total
number of parameters, confounding depth and size, we construct three classes of
models which trade off depth for width such that the total number of parameters
is kept constant (41M, 134M and 374M parameters). We pretrain all models as LMs
and fine-tune them on tasks that test for compositional generalization. We
report three main conclusions: (1) after fine-tuning, deeper models generalize
better out-of-distribution than shallower models do, but the relative benefit
of additional layers diminishes rapidly; (2) within each family, deeper models
show better language modeling performance, but returns are similarly
diminishing; (3) the benefits of depth for compositional generalization cannot
be attributed solely to better performance on language modeling or on
in-distribution data.
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models
  across Computer Vision Tasks
Authors: name: Micah Goldblum, name: Hossein Souri, name: Renkun Ni, name: Manli Shu, name: Viraj Prabhu, name: Gowthami Somepalli, name: Prithvijit Chattopadhyay, name: Mark Ibrahim, name: Adrien Bardes, name: Judy Hoffman, name: Rama Chellappa, name: Andrew Gordon Wilson, name: Tom Goldstein
Published: 2023-10-30 18:23:58

Neural network based computer vision systems are typically built on a
backbone, a pretrained or randomly initialized feature extractor. Several years
ago, the default option was an ImageNet-trained convolutional neural network.
However, the recent past has seen the emergence of countless backbones
pretrained using various algorithms and datasets. While this abundance of
choice has led to performance increases for a range of systems, it is difficult
for practitioners to make informed decisions about which backbone to choose.
Battle of the Backbones (BoB) makes this choice easier by benchmarking a
diverse suite of pretrained models, including vision-language models, those
trained via self-supervised learning, and the Stable Diffusion backbone, across
a diverse set of computer vision tasks ranging from classification to object
detection to OOD generalization and more. Furthermore, BoB sheds light on
promising directions for the research community to advance computer vision by
illuminating strengths and weakness of existing approaches through a
comprehensive analysis conducted on more than 1500 training runs. While vision
transformers (ViTs) and self-supervised learning (SSL) are increasingly
popular, we find that convolutional neural networks pretrained in a supervised
fashion on large training sets still perform best on most tasks among the
models we consider. Moreover, in apples-to-apples comparisons on the same
architectures and similarly sized pretraining datasets, we find that SSL
backbones are highly competitive, indicating that future works should perform
SSL pretraining with advanced architectures and larger pretraining datasets. We
release the raw results of our experiments along with code that allows
researchers to put their own backbones through the gauntlet here:
https://github.com/hsouri/Battle-of-the-Backbones
Learning From Mistakes Makes LLM Better Reasoner
Authors: name: Shengnan An, name: Zexiong Ma, name: Zeqi Lin, name: Nanning Zheng, name: Jian-Guang Lou, name: Weizhu Chen
Published: 2023-10-31 17:52:22

Large language models (LLMs) recently exhibited remarkable reasoning
capabilities on solving math problems. To further improve this capability, this
work proposes Learning from Mistakes (LeMa), akin to human learning processes.
Consider a human student who failed to solve a math problem, he will learn from
what mistake he has made and how to correct it. Mimicking this error-driven
learning process, LeMa fine-tunes LLMs on mistake-correction data pairs
generated by GPT-4. Specifically, we first collect inaccurate reasoning paths
from various LLMs and then employ GPT-4 as a "corrector" to (1) identify the
mistake step, (2) explain the reason for the mistake, and (3) correct the
mistake and generate the final answer. Experimental results demonstrate the
effectiveness of LeMa: across five backbone LLMs and two mathematical reasoning
tasks, LeMa consistently improves the performance compared with fine-tuning on
CoT data alone. Impressively, LeMa can also benefit specialized LLMs such as
WizardMath and MetaMath, achieving 85.4% pass@1 accuracy on GSM8K and 27.1% on
MATH. This surpasses the SOTA performance achieved by non-execution open-source
models on these challenging tasks. Our code, data and models will be publicly
available at https://github.com/microsoft/CodeT.
Multimodal ChatGPT for Medical Applications: an Experimental Study of
  GPT-4V
Authors: name: Zhiling Yan, name: Kai Zhang, name: Rong Zhou, name: Lifang He, name: Xiang Li, name: Lichao Sun
Published: 2023-10-29 16:26:28

In this paper, we critically evaluate the capabilities of the
state-of-the-art multimodal large language model, i.e., GPT-4 with Vision
(GPT-4V), on Visual Question Answering (VQA) task. Our experiments thoroughly
assess GPT-4V's proficiency in answering questions paired with images using
both pathology and radiology datasets from 11 modalities (e.g. Microscopy,
Dermoscopy, X-ray, CT, etc.) and fifteen objects of interests (brain, liver,
lung, etc.). Our datasets encompass a comprehensive range of medical inquiries,
including sixteen distinct question types. Throughout our evaluations, we
devised textual prompts for GPT-4V, directing it to synergize visual and
textual information. The experiments with accuracy score conclude that the
current version of GPT-4V is not recommended for real-world diagnostics due to
its unreliable and suboptimal accuracy in responding to diagnostic medical
questions. In addition, we delineate seven unique facets of GPT-4V's behavior
in medical VQA, highlighting its constraints within this complex arena. The
complete details of our evaluation cases are accessible at
https://github.com/ZhilingYan/GPT4V-Medical-Report.
Text-to-3D with classifier score distillation
Authors: name: Xin Yu, name: Yuan-Chen Guo, name: Yangguang Li, name: Ding Liang, name: Song-Hai Zhang, name: Xiaojuan Qi
Published: 2023-10-30 10:25:40

Text-to-3D generation has made remarkable progress recently, particularly
with methods based on Score Distillation Sampling (SDS) that leverages
pre-trained 2D diffusion models. While the usage of classifier-free guidance is
well acknowledged to be crucial for successful optimization, it is considered
an auxiliary trick rather than the most essential component. In this paper, we
re-evaluate the role of classifier-free guidance in score distillation and
discover a surprising finding: the guidance alone is enough for effective
text-to-3D generation tasks. We name this method Classifier Score Distillation
(CSD), which can be interpreted as using an implicit classification model for
generation. This new perspective reveals new insights for understanding
existing techniques. We validate the effectiveness of CSD across a variety of
text-to-3D tasks including shape generation, texture synthesis, and shape
editing, achieving results superior to those of state-of-the-art methods. Our
project page is https://xinyu-andy.github.io/Classifier-Score-Distillation
VideoCrafter1: Open Diffusion Models for High-Quality Video Generation
Authors: name: Haoxin Chen, name: Menghan Xia, name: Yingqing He, name: Yong Zhang, name: Xiaodong Cun, name: Shaoshu Yang, name: Jinbo Xing, name: Yaofang Liu, name: Qifeng Chen, name: Xintao Wang, name: Chao Weng, name: Ying Shan
Published: 2023-10-30 13:12:40

Video generation has increasingly gained interest in both academia and
industry. Although commercial tools can generate plausible videos, there is a
limited number of open-source models available for researchers and engineers.
In this work, we introduce two diffusion models for high-quality video
generation, namely text-to-video (T2V) and image-to-video (I2V) models. T2V
models synthesize a video based on a given text input, while I2V models
incorporate an additional image input. Our proposed T2V model can generate
realistic and cinematic-quality videos with a resolution of 1024 times 576,
outperforming other open-source T2V models in terms of quality. The I2V model
is designed to produce videos that strictly adhere to the content of the
provided reference image, preserving its content, structure, and style. This
model is the first open-source I2V foundation model capable of transforming a
given image into a video clip while maintaining content preservation
constraints. We believe that these open-source video generation models will
contribute significantly to the technological advancements within the
community.
CustomNet: Zero-shot Object Customization with Variable-Viewpoints in
  Text-to-Image Diffusion Models
Authors: name: Ziyang Yuan, name: Mingdeng Cao, name: Xintao Wang, name: Zhongang Qi, name: Chun Yuan, name: Ying Shan
Published: 2023-10-30 17:50:14

Incorporating a customized object into image generation presents an
attractive feature in text-to-image generation. However, existing
optimization-based and encoder-based methods are hindered by drawbacks such as
time-consuming optimization, insufficient identity preservation, and a
prevalent copy-pasting effect. To overcome these limitations, we introduce
CustomNet, a novel object customization approach that explicitly incorporates
3D novel view synthesis capabilities into the object customization process.
This integration facilitates the adjustment of spatial position relationships
and viewpoints, yielding diverse outputs while effectively preserving object
identity. Moreover, we introduce delicate designs to enable location control
and flexible background control through textual descriptions or specific
user-defined images, overcoming the limitations of existing 3D novel view
synthesis methods. We further leverage a dataset construction pipeline that can
better handle real-world objects and complex backgrounds. Equipped with these
designs, our method facilitates zero-shot object customization without
test-time optimization, offering simultaneous control over the viewpoints,
location, and background. As a result, our CustomNet ensures enhanced identity
preservation and generates diverse, harmonious outputs.
Skywork: A More Open Bilingual Foundation Model
Authors: name: Tianwen Wei, name: Liang Zhao, name: Lichang Zhang, name: Bo Zhu, name: Lijie Wang, name: Haihua Yang, name: Biye Li, name: Cheng Cheng, name: Weiwei Lü, name: Rui Hu, name: Chenxia Li, name: Liu Yang, name: Xilin Luo, name: Xuejie Wu, name: Lunan Liu, name: Wenjun Cheng, name: Peng Cheng, name: Jianhao Zhang, name: Xiaoyu Zhang, name: Lei Lin, name: Xiaokun Wang, name: Yutuan Ma, name: Chuanhai Dong, name: Yanqi Sun, name: Yifu Chen, name: Yongyi Peng, name: Xiaojuan Liang, name: Shuicheng Yan, name: Han Fang, name: Yahui Zhou
Published: 2023-10-30 08:31:47

In this technical report, we present Skywork-13B, a family of large language
models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both
English and Chinese texts. This bilingual foundation model is the most
extensively trained and openly published LLMs of comparable size to date. We
introduce a two-stage training methodology using a segmented corpus, targeting
general purpose training and then domain-specific enhancement training,
respectively. We show that our model not only excels on popular benchmarks, but
also achieves state of the art performance in Chinese language modeling
on diverse domains. Furthermore, we propose a novel leakage detection method,
demonstrating that test data contamination is a pressing issue warranting
further investigation by the LLM community. To spur future research, we release
Skywork-13B along with checkpoints obtained during intermediate stages of the
training process. We are also releasing part of our SkyPile corpus, a
collection of over 150 billion tokens of web text, which is the largest high
quality open Chinese pre-training corpus to date. We hope Skywork-13B and our
open corpus will serve as a valuable open-source resource to democratize access
to high-quality LLMs.