Orionnnn/hub-stats huggingface dataset:
MagicProp: Diffusion-based Video Editing via Motion-aware Appearance
  Propagation
Authors: name: Hanshu Yan, name: Jun Hao Liew, name: Long Mai, name: Shanchuan Lin, name: Jiashi Feng
Published: 2023-09-02 11:13:29

This paper addresses the issue of modifying the visual appearance of videos
while preserving their motion. A novel framework, named MagicProp, is proposed,
which disentangles the video editing process into two stages: appearance
editing and motion-aware appearance propagation. In the first stage, MagicProp
selects a single frame from the input video and applies image-editing
techniques to modify the content and/or style of the frame. The flexibility of
these techniques enables the editing of arbitrary regions within the frame. In
the second stage, MagicProp employs the edited frame as an appearance reference
and generates the remaining frames using an autoregressive rendering approach.
To achieve this, a diffusion-based conditional generation model, called
PropDPM, is developed, which synthesizes the target frame by conditioning on
the reference appearance, the target motion, and its previous appearance. The
autoregressive editing approach ensures temporal consistency in the resulting
videos. Overall, MagicProp combines the flexibility of image-editing techniques
with the superior temporal consistency of autoregressive modeling, enabling
flexible editing of object types and aesthetic styles in arbitrary regions of
input videos while maintaining good temporal consistency across frames.
Extensive experiments in various video editing scenarios demonstrate the
effectiveness of MagicProp.
StyleAdapter: A Single-Pass LoRA-Free Model for Stylized Image
  Generation
Authors: name: Zhouxia Wang, name: Xintao Wang, name: Liangbin Xie, name: Zhongang Qi, name: Ying Shan, name: Wenping Wang, name: Ping Luo
Published: 2023-09-04 19:16:46

This paper presents a LoRA-free method for stylized image generation that
takes a text prompt and style reference images as inputs and produces an output
image in a single pass. Unlike existing methods that rely on training a
separate LoRA for each style, our method can adapt to various styles with a
unified model. However, this poses two challenges: 1) the prompt loses
controllability over the generated content, and 2) the output image inherits
both the semantic and style features of the style reference image, compromising
its content fidelity. To address these challenges, we introduce StyleAdapter, a
model that comprises two components: a two-path cross-attention module (TPCA)
and three decoupling strategies. These components enable our model to process
the prompt and style reference features separately and reduce the strong
coupling between the semantic and style information in the style references.
StyleAdapter can generate high-quality images that match the content of the
prompts and adopt the style of the references (even for unseen styles) in a
single pass, which is more flexible and efficient than previous methods.
Experiments have been conducted to demonstrate the superiority of our method
over previous works.
PromptTTS 2: Describing and Generating Voices with Text Prompt
Authors: name: Yichong Leng, name: Zhifang Guo, name: Kai Shen, name: Xu Tan, name: Zeqian Ju, name: Yanqing Liu, name: Yufei Liu, name: Dongchao Yang, name: Leying Zhang, name: Kaitao Song, name: Lei He, name: Xiang-Yang Li, name: Sheng Zhao, name: Tao Qin, name: Jiang Bian
Published: 2023-09-05 14:45:27

Speech conveys more information than just text, as the same word can be
uttered in various voices to convey diverse information. Compared to
traditional text-to-speech (TTS) methods relying on speech prompts (reference
speech) for voice variability, using text prompts (descriptions) is more
user-friendly since speech prompts can be hard to find or may not exist at all.
TTS approaches based on the text prompt face two challenges: 1) the one-to-many
problem, where not all details about voice variability can be described in the
text prompt, and 2) the limited availability of text prompt datasets, where
vendors and large cost of data labeling are required to write text prompt for
speech. In this work, we introduce PromptTTS 2 to address these challenges with
a variation network to provide variability information of voice not captured by
text prompts, and a prompt generation pipeline to utilize the large language
models (LLM) to compose high quality text prompts. Specifically, the variation
network predicts the representation extracted from the reference speech (which
contains full information about voice) based on the text prompt representation.
For the prompt generation pipeline, it generates text prompts for speech with a
speech understanding model to recognize voice attributes (e.g., gender, speed)
from speech and a large language model to formulate text prompt based on the
recognition results. Experiments on a large-scale (44K hours) speech dataset
demonstrate that compared to the previous works, PromptTTS 2 generates voices
more consistent with text prompts and supports the sampling of diverse voice
variability, thereby offering users more choices on voice generation.
Additionally, the prompt generation pipeline produces high-quality prompts,
eliminating the large labeling cost. The demo page of PromptTTS 2 is available
onlinehttps://speechresearch.github.io/prompttts2.
Efficient RLHF: Reducing the Memory Usage of PPO
Authors: name: Michael Santacroce, name: Yadong Lu, name: Han Yu, name: Yuanzhi Li, name: Yelong Shen
Published: 2023-09-01 22:57:20

Reinforcement Learning with Human Feedback (RLHF) has revolutionized language
modeling by aligning models with human preferences. However, the RL stage,
Proximal Policy Optimization (PPO), requires over 3x the memory of Supervised
Fine-Tuning (SFT), making it infeasible to use for most practitioners. To
address this issue, we present a comprehensive analysis the memory usage,
performance, and training time of memory-savings techniques for PPO. We
introduce Hydra-RLHF by first integrating the SFT and Reward models and then
dynamically turning LoRA "off" during training. Our experiments show: 1. Using
LoRA during PPO reduces its memory usage to be smaller than SFT while improving
alignment across four public benchmarks, and 2. Hydra-PPO reduces the latency
per sample of LoRA-PPO by up to 65% while maintaining its performance. Our
results demonstrate that Hydra-PPO is a simple and promising solution for
enabling more widespread usage of RLHF.
ModelScope-Agent: Building Your Customizable Agent System with
  Open-source Large Language Models
Authors: name: Chenliang Li, name: Hehong Chen, name: Ming Yan, name: Weizhou Shen, name: Haiyang Xu, name: Zhikai Wu, name: Zhicheng Zhang, name: Wenmeng Zhou, name: Yingda Chen, name: Chen Cheng, name: Hongzhu Shi, name: Ji Zhang, name: Fei Huang, name: Jingren Zhou
Published: 2023-09-02 16:50:30

Large language models (LLMs) have recently demonstrated remarkable
capabilities to comprehend human intentions, engage in reasoning, and design
planning-like behavior. To further unleash the power of LLMs to accomplish
complex tasks, there is a growing trend to build agent framework that equips
LLMs, such as ChatGPT, with tool-use abilities to connect with massive external
APIs. In this work, we introduce ModelScope-Agent, a general and customizable
agent framework for real-world applications, based on open-source LLMs as
controllers. It provides a user-friendly system library, with customizable
engine design to support model training on multiple open-source LLMs, while
also enabling seamless integration with both model APIs and common APIs in a
unified way. To equip the LLMs with tool-use abilities, a comprehensive
framework has been proposed spanning over tool-use data collection, tool
retrieval, tool registration, memory control, customized model training, and
evaluation for practical real-world applications. Finally, we showcase
ModelScopeGPT, a real-world intelligent assistant of ModelScope Community based
on the ModelScope-Agent framework, which is able to connect open-source LLMs
with more than 1000 public AI models and localized community knowledge in
ModelScope. The ModelScope-Agent
libraryhttps://github.com/modelscope/modelscope-agent and online
demohttps://modelscope.cn/studios/damo/ModelScopeGPT/summary are now
publicly available.
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D
  Understanding, Generation, and Instruction Following
Authors: name: Ziyu Guo, name: Renrui Zhang, name: Xiangyang Zhu, name: Yiwen Tang, name: Xianzheng Ma, name: Jiaming Han, name: Kexin Chen, name: Peng Gao, name: Xianzhi Li, name: Hongsheng Li, name: Pheng-Ann Heng
Published: 2023-09-01 17:59:47

We introduce Point-Bind, a 3D multi-modality model aligning point clouds with
2D image, language, audio, and video. Guided by ImageBind, we construct a joint
embedding space between 3D and multi-modalities, enabling many promising
applications, e.g., any-to-3D generation, 3D embedding arithmetic, and 3D
open-world understanding. On top of this, we further present Point-LLM, the
first 3D large language model (LLM) following 3D multi-modal instructions. By
parameter-efficient fine-tuning techniques, Point-LLM injects the semantics of
Point-Bind into pre-trained LLMs, e.g., LLaMA, which requires no 3D instruction
data, but exhibits superior 3D and multi-modal question-answering capacity. We
hope our work may cast a light on the community for extending 3D point clouds
to multi-modality applications. Code is available at
https://github.com/ZiyuGuo99/Point-Bind_Point-LLM.
CityDreamer: Compositional Generative Model of Unbounded 3D Cities
Authors: name: Haozhe Xie, name: Zhaoxi Chen, name: Fangzhou Hong, name: Ziwei Liu
Published: 2023-09-01 17:57:02

In recent years, extensive research has focused on 3D natural scene
generation, but the domain of 3D city generation has not received as much
exploration. This is due to the greater challenges posed by 3D city generation,
mainly because humans are more sensitive to structural distortions in urban
environments. Additionally, generating 3D cities is more complex than 3D
natural scenes since buildings, as objects of the same class, exhibit a wider
range of appearances compared to the relatively consistent appearance of
objects like trees in natural scenes. To address these challenges, we propose
CityDreamer, a compositional generative model designed specifically for
unbounded 3D cities, which separates the generation of building instances from
other background objects, such as roads, green lands, and water areas, into
distinct modules. Furthermore, we construct two datasets, OSM and GoogleEarth,
containing a vast amount of real-world city imagery to enhance the realism of
the generated 3D cities both in their layouts and appearances. Through
extensive experiments, CityDreamer has proven its superiority over
state-of-the-art methods in generating a wide range of lifelike 3D cities.
FACET: Fairness in Computer Vision Evaluation Benchmark
Authors: name: Laura Gustafson, name: Chloe Rolland, name: Nikhila Ravi, name: Quentin Duval, name: Aaron Adcock, name: Cheng-Yang Fu, name: Melissa Hall, name: Candace Ross
Published: 2023-08-31 17:59:48

Computer vision models have known performance disparities across attributes
such as gender and skin tone. This means during tasks such as classification
and detection, model performance differs for certain classes based on the
demographics of the people in the image. These disparities have been shown to
exist, but until now there has not been a unified approach to measure these
differences for common use-cases of computer vision models. We present a new
benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large,
publicly available evaluation set of 32k images for some of the most common
vision tasks - image classification, object detection and segmentation. For
every image in FACET, we hired expert reviewers to manually annotate
person-related attributes such as perceived skin tone and hair type, manually
draw bounding boxes and label fine-grained person-related classes such as disk
jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art
vision models and present a deeper understanding of potential performance
disparities and challenges across sensitive demographic attributes. With the
exhaustive annotations collected, we probe models using single demographics
attributes as well as multiple attributes using an intersectional approach
(e.g. hair color and perceived skin tone). Our results show that
classification, detection, segmentation, and visual grounding models exhibit
performance disparities across demographic attributes and intersections of
attributes. These harms suggest that not all people represented in datasets
receive fair and equitable treatment in these vision tasks. We hope current and
future results using our benchmark will contribute to fairer, more robust
vision models. FACET is available publicly at https://facet.metademolab.com/
VideoGen: A Reference-Guided Latent Diffusion Approach for High
  Definition Text-to-Video Generation
Authors: name: Xin Li, name: Wenqing Chu, name: Ye Wu, name: Weihang Yuan, name: Fanglong Liu, name: Qi Zhang, name: Fu Li, name: Haocheng Feng, name: Errui Ding, name: Jingdong Wang
Published: 2023-09-01 11:14:43

In this paper, we present VideoGen, a text-to-video generation approach,
which can generate a high-definition video with high frame fidelity and strong
temporal consistency using reference-guided latent diffusion. We leverage an
off-the-shelf text-to-image generation model, e.g., Stable Diffusion, to
generate an image with high content quality from the text prompt, as a
reference image to guide video generation. Then, we introduce an efficient
cascaded latent diffusion module conditioned on both the reference image and
the text prompt, for generating latent video representations, followed by a
flow-based temporal upsampling step to improve the temporal resolution.
Finally, we map latent video representations into a high-definition video
through an enhanced video decoder. During training, we use the first frame of a
ground-truth video as the reference image for training the cascaded latent
diffusion module. The main characterises of our approach include: the reference
image generated by the text-to-image model improves the visual fidelity; using
it as the condition makes the diffusion model focus more on learning the video
dynamics; and the video decoder is trained over unlabeled video data, thus
benefiting from high-quality easily-available videos. VideoGen sets a new
state-of-the-art in text-to-video generation in terms of both qualitative and
quantitative evaluation.
Large Content And Behavior Models To Understand, Simulate, And Optimize
  Content And Behavior
Authors: name: Ashmit Khandelwal, name: Aditya Agrawal, name: Aanisha Bhattacharyya, name: Yaman K Singla, name: Somesh Singh, name: Uttaran Bhattacharya, name: Ishita Dasgupta, name: Stefano Petrangeli, name: Rajiv Ratn Shah, name: Changyou Chen, name: Balaji Krishnamurthy
Published: 2023-09-01 09:34:49

Shannon, in his seminal paper introducing information theory, divided the
communication into three levels: technical, semantic, and effectivenss. While
the technical level is concerned with accurate reconstruction of transmitted
symbols, the semantic and effectiveness levels deal with the inferred meaning
and its effect on the receiver. Thanks to telecommunications, the first level
problem has produced great advances like the internet. Large Language Models
(LLMs) make some progress towards the second goal, but the third level still
remains largely untouched. The third problem deals with predicting and
optimizing communication for desired receiver behavior. LLMs, while showing
wide generalization capabilities across a wide range of tasks, are unable to
solve for this. One reason for the underperformance could be a lack of
"behavior tokens" in LLMs' training corpora. Behavior tokens define receiver
behavior over a communication, such as shares, likes, clicks, purchases,
retweets, etc. While preprocessing data for LLM training, behavior tokens are
often removed from the corpora as noise. Therefore, in this paper, we make some
initial progress towards reintroducing behavior tokens in LLM training. The
trained models, other than showing similar performance to LLMs on content
understanding tasks, show generalization capabilities on behavior simulation,
content simulation, behavior understanding, and behavior domain adaptation.
Using a wide range of tasks on two corpora, we show results on all these
capabilities. We call these models Large Content and Behavior Models (LCBMs).
Further, to spur more research on LCBMs, we release our new Content Behavior
Corpus (CBC), a repository containing communicator, message, and corresponding
receiver behavior.
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI
  Feedback
Authors: name: Harrison Lee, name: Samrat Phatale, name: Hassan Mansoor, name: Kellie Lu, name: Thomas Mesnard, name: Colton Bishop, name: Victor Carbune, name: Abhinav Rastogi
Published: 2023-09-01 05:53:33

Reinforcement learning from human feedback (RLHF) is effective at aligning
large language models (LLMs) to human preferences, but gathering high quality
human preference labels is a key bottleneck. We conduct a head-to-head
comparison of RLHF vs. RL from AI Feedback (RLAIF) - a technique where
preferences are labeled by an off-the-shelf LLM in lieu of humans, and we find
that they result in similar improvements. On the task of summarization, human
evaluators prefer generations from both RLAIF and RLHF over a baseline
supervised fine-tuned model in ~70% of cases. Furthermore, when asked to rate
RLAIF vs. RLHF summaries, humans prefer both at equal rates. These results
suggest that RLAIF can yield human-level performance, offering a potential
solution to the scalability limitations of RLHF.
YaRN: Efficient Context Window Extension of Large Language Models
Authors: name: Bowen Peng, name: Jeffrey Quesnelle, name: Honglu Fan, name: Enrico Shippole
Published: 2023-08-31 18:18:07

Rotary Position Embeddings (RoPE) have been shown to effectively encode
positional information in transformer-based language models. However, these
models fail to generalize past the sequence length they were trained on. We
present YaRN (Yet another RoPE extensioN method), a compute-efficient method to
extend the context window of such models, requiring 10x less tokens and 2.5x
less training steps than previous methods. Using YaRN, we show that LLaMA
models can effectively utilize and extrapolate to context lengths much longer
than their original pre-training would allow, while also surpassing previous
the state-of-the-art at context window extension. In addition, we demonstrate
that YaRN exhibits the capability to extrapolate beyond the limited context of
a fine-tuning dataset. We publish the checkpoints of Llama 2 7B/13B fine-tuned
using YaRN with 64k and 128k context windows at
https://github.com/jquesnelle/yarn
SportsSloMo: A New Benchmark and Baselines for Human-centric Video Frame
  Interpolation
Authors: name: Jiaben Chen, name: Huaizu Jiang
Published: 2023-08-31 17:23:50

Human-centric video frame interpolation has great potential for improving
people's entertainment experiences and finding commercial applications in the
sports analysis industry, e.g., synthesizing slow-motion videos. Although there
are multiple benchmark datasets available in the community, none of them is
dedicated for human-centric scenarios. To bridge this gap, we introduce
SportsSloMo, a benchmark consisting of more than 130K video clips and 1M video
frames of high-resolution (geq720p) slow-motion sports videos crawled from
YouTube. We re-train several state-of-the-art methods on our benchmark, and the
results show a decrease in their accuracy compared to other datasets. It
highlights the difficulty of our benchmark and suggests that it poses
significant challenges even for the best-performing methods, as human bodies
are highly deformable and occlusions are frequent in sports videos. To improve
the accuracy, we introduce two loss terms considering the human-aware priors,
where we add auxiliary supervision to panoptic segmentation and human keypoints
detection, respectively. The loss terms are model agnostic and can be easily
plugged into any video frame interpolation approaches. Experimental results
validate the effectiveness of our proposed loss terms, leading to consistent
performance improvement over 5 existing models, which establish strong baseline
models on our benchmark. The dataset and code can be found at:
https://neu-vi.github.io/SportsSlomo/.
Active Neural Mapping
Authors: name: Zike Yan, name: Haoxiang Yang, name: Hongbin Zha
Published: 2023-08-30 18:07:30

We address the problem of active mapping with a continually-learned neural
scene representation, namely Active Neural Mapping. The key lies in actively
finding the target space to be explored with efficient agent movement, thus
minimizing the map uncertainty on-the-fly within a previously unseen
environment. In this paper, we examine the weight space of the
continually-learned neural field, and show empirically that the neural
variability, the prediction robustness against random weight perturbation, can
be directly utilized to measure the instant uncertainty of the neural map.
Together with the continuous geometric information inherited in the neural map,
the agent can be guided to find a traversable path to gradually gain knowledge
of the environment. We present for the first time an active mapping system with
a coordinate-based implicit neural representation for online scene
reconstruction. Experiments in the visually-realistic Gibson and Matterport3D
environment demonstrate the efficacy of the proposed method.
Emergence of Segmentation with Minimalistic White-Box Transformers
Authors: name: Yaodong Yu, name: Tianzhe Chu, name: Shengbang Tong, name: Ziyang Wu, name: Druv Pai, name: Sam Buchanan, name: Yi Ma
Published: 2023-08-30 19:02:17

Transformer-like models for vision tasks have recently proven effective for a
wide range of downstream applications such as segmentation and detection.
Previous works have shown that segmentation properties emerge in vision
transformers (ViTs) trained using self-supervised methods such as DINO, but not
in those trained on supervised classification tasks. In this study, we probe
whether segmentation emerges in transformer-based models solely as a result of
intricate self-supervised learning mechanisms, or if the same emergence can be
achieved under much broader conditions through proper design of the model
architecture. Through extensive experimental results, we demonstrate that when
employing a white-box transformer-like architecture known as CRATE, whose
design explicitly models and pursues low-dimensional structures in the data
distribution, segmentation properties, at both the whole and parts levels,
already emerge with a minimalistic supervised training recipe. Layer-wise
finer-grained analysis reveals that the emergent properties strongly
corroborate the designed mathematical functions of the white-box network. Our
results suggest a path to design white-box foundation models that are
simultaneously highly performant and mathematically fully interpretable. Code
is at https://github.com/Ma-Lab-Berkeley/CRATE.