Orionnnn/hub-stats huggingface dataset:
DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule
  Graphs
Authors: name: Youwei Liang, name: Ruiyi Zhang, name: Li Zhang, name: Pengtao Xie
Published: 2023-05-18 16:22:33

A ChatGPT-like system for drug compounds could be a game-changer in
pharmaceutical research, accelerating drug discovery, enhancing our
understanding of structure-activity relationships, guiding lead optimization,
aiding drug repurposing, reducing the failure rate, and streamlining clinical
trials. In this work, we make an initial attempt towards enabling ChatGPT-like
capabilities on drug molecule graphs, by developing a prototype system
DrugChat. DrugChat works in a similar way as ChatGPT. Users upload a compound
molecule graph and ask various questions about this compound. DrugChat will
answer these questions in a multi-turn, interactive manner. The DrugChat system
consists of a graph neural network (GNN), a large language model (LLM), and an
adaptor. The GNN takes a compound molecule graph as input and learns a
representation for this graph. The adaptor transforms the graph representation
produced by the GNN into another representation that is acceptable to the LLM.
The LLM takes the compound representation transformed by the adaptor and users'
questions about this compound as inputs and generates answers. All these
components are trained end-to-end. To train DrugChat, we collected instruction
tuning datasets which contain 10,834 drug compounds and 143,517 question-answer
pairs. The code and data is available at
https://github.com/UCSD-AI4H/drugchat
From Sparse to Dense: GPT-4 Summarization with Chain of Density
  Prompting
Authors: name: Griffin Adams, name: Alexander Fabbri, name: Faisal Ladhak, name: Eric Lehman, name: Noémie Elhadad
Published: 2023-09-08 11:31:08

Selecting the ``right'' amount of information to include in a summary is a
difficult task. A good summary should be detailed and entity-centric without
being overly dense and hard to follow. To better understand this tradeoff, we
solicit increasingly dense GPT-4 summaries with what we refer to as a ``Chain
of Density'' (CoD) prompt. Specifically, GPT-4 generates an initial
entity-sparse summary before iteratively incorporating missing salient entities
without increasing the length. Summaries generated by CoD are more abstractive,
exhibit more fusion, and have less of a lead bias than GPT-4 summaries
generated by a vanilla prompt. We conduct a human preference study on 100 CNN
DailyMail articles and find that that humans prefer GPT-4 summaries that are
more dense than those generated by a vanilla prompt and almost as dense as
human written summaries. Qualitative analysis supports the notion that there
exists a tradeoff between informativeness and readability. 500 annotated CoD
summaries, as well as an extra 5,000 unannotated summaries, are freely
available on HuggingFace
(https://huggingface.co/datasets/griffin/chain_of_density).
Tracking Anything with Decoupled Video Segmentation
Authors: name: Ho Kei Cheng, name: Seoung Wug Oh, name: Brian Price, name: Alexander Schwing, name: Joon-Young Lee
Published: 2023-09-07 17:59:41

Training data for video segmentation are expensive to annotate. This impedes
extensions of end-to-end algorithms to new video segmentation tasks, especially
in large-vocabulary settings. To 'track anything' without training on video
data for every individual task, we develop a decoupled video segmentation
approach (DEVA), composed of task-specific image-level segmentation and
class/task-agnostic bi-directional temporal propagation. Due to this design, we
only need an image-level model for the target task (which is cheaper to train)
and a universal temporal propagation model which is trained once and
generalizes across tasks. To effectively combine these two modules, we use
bi-directional propagation for (semi-)online fusion of segmentation hypotheses
from different frames to generate a coherent segmentation. We show that this
decoupled formulation compares favorably to end-to-end approaches in several
data-scarce tasks including large-vocabulary video panoptic segmentation,
open-world video segmentation, referring video segmentation, and unsupervised
video object segmentation. Code is available at:
https://hkchengrex.github.io/Tracking-Anything-with-DEVA
Robotic Table Tennis: A Case Study into a High Speed Learning System
Authors: name: David B. D'Ambrosio, name: Jonathan Abelian, name: Saminda Abeyruwan, name: Michael Ahn, name: Alex Bewley, name: Justin Boyd, name: Krzysztof Choromanski, name: Omar Cortes, name: Erwin Coumans, name: Tianli Ding, name: Wenbo Gao, name: Laura Graesser, name: Atil Iscen, name: Navdeep Jaitly, name: Deepali Jain, name: Juhana Kangaspunta, name: Satoshi Kataoka, name: Gus Kouretas, name: Yuheng Kuang, name: Nevena Lazic, name: Corey Lynch, name: Reza Mahjourian, name: Sherry Q. Moore, name: Thinh Nguyen, name: Ken Oslund, name: Barney J Reed, name: Krista Reymann, name: Pannag R. Sanketi, name: Anish Shankar, name: Pierre Sermanet, name: Vikas Sindhwani, name: Avi Singh, name: Vincent Vanhoucke, name: Grace Vesom, name: Peng Xu
Published: 2023-09-06 18:56:20

We present a deep-dive into a real-world robotic learning system that, in
previous work, was shown to be capable of hundreds of table tennis rallies with
a human and has the ability to precisely return the ball to desired targets.
This system puts together a highly optimized perception subsystem, a high-speed
low-latency robot controller, a simulation paradigm that can prevent damage in
the real world and also train policies for zero-shot transfer, and automated
real world environment resets that enable autonomous training and evaluation on
physical robots. We complement a complete system description, including
numerous design decisions that are typically not widely disseminated, with a
collection of studies that clarify the importance of mitigating various sources
of latency, accounting for training and deployment distribution shifts,
robustness of the perception system, sensitivity to policy hyper-parameters,
and choice of action space. A video demonstrating the components of the system
and details of experimental results can be found at
https://youtu.be/uFcnWjB42I0.
ProPainter: Improving Propagation and Transformer for Video Inpainting
Authors: name: Shangchen Zhou, name: Chongyi Li, name: Kelvin C. K. Chan, name: Chen Change Loy
Published: 2023-09-07 17:57:29

Flow-based propagation and spatiotemporal Transformer are two mainstream
mechanisms in video inpainting (VI). Despite the effectiveness of these
components, they still suffer from some limitations that affect their
performance. Previous propagation-based approaches are performed separately
either in the image or feature domain. Global image propagation isolated from
learning may cause spatial misalignment due to inaccurate optical flow.
Moreover, memory or computational constraints limit the temporal range of
feature propagation and video Transformer, preventing exploration of
correspondence information from distant frames. To address these issues, we
propose an improved framework, called ProPainter, which involves enhanced
ProPagation and an efficient Transformer. Specifically, we introduce
dual-domain propagation that combines the advantages of image and feature
warping, exploiting global correspondences reliably. We also propose a
mask-guided sparse video Transformer, which achieves high efficiency by
discarding unnecessary and redundant tokens. With these components, ProPainter
outperforms prior arts by a large margin of 1.46 dB in PSNR while maintaining
appealing efficiency.
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks
Authors: name: Zigang Geng, name: Binxin Yang, name: Tiankai Hang, name: Chen Li, name: Shuyang Gu, name: Ting Zhang, name: Jianmin Bao, name: Zheng Zhang, name: Han Hu, name: Dong Chen, name: Baining Guo
Published: 2023-09-07 17:56:57

We present InstructDiffusion, a unifying and generic framework for aligning
computer vision tasks with human instructions. Unlike existing approaches that
integrate prior knowledge and pre-define the output space (e.g., categories and
coordinates) for each vision task, we cast diverse vision tasks into a
human-intuitive image-manipulating process whose output space is a flexible and
interactive pixel space. Concretely, the model is built upon the diffusion
process and is trained to predict pixels according to user instructions, such
as encircling the man's left shoulder in red or applying a blue mask to the
left car. InstructDiffusion could handle a variety of vision tasks, including
understanding tasks (such as segmentation and keypoint detection) and
generative tasks (such as editing and enhancement). It even exhibits the
ability to handle unseen tasks and outperforms prior methods on novel datasets.
This represents a significant step towards a generalist modeling interface for
vision tasks, advancing artificial general intelligence in the field of
computer vision.
SyncDreamer: Generating Multiview-consistent Images from a Single-view
  Image
Authors: name: Yuan Liu, name: Cheng Lin, name: Zijiao Zeng, name: Xiaoxiao Long, name: Lingjie Liu, name: Taku Komura, name: Wenping Wang
Published: 2023-09-07 02:28:04

In this paper, we present a novel diffusion model called that generates
multiview-consistent images from a single-view image. Using pretrained
large-scale 2D diffusion models, recent work Zero123 demonstrates the ability
to generate plausible novel views from a single-view image of an object.
However, maintaining consistency in geometry and colors for the generated
images remains a challenge. To address this issue, we propose a synchronized
multiview diffusion model that models the joint probability distribution of
multiview images, enabling the generation of multiview-consistent images in a
single reverse process. SyncDreamer synchronizes the intermediate states of all
the generated images at every step of the reverse process through a 3D-aware
feature attention mechanism that correlates the corresponding features across
different views. Experiments show that SyncDreamer generates images with high
consistency across different views, thus making it well-suited for various 3D
generation tasks such as novel-view-synthesis, text-to-3D, and image-to-3D.
Reuse and Diffuse: Iterative Denoising for Text-to-Video Generation
Authors: name: Jiaxi Gu, name: Shicong Wang, name: Haoyu Zhao, name: Tianyi Lu, name: Xing Zhang, name: Zuxuan Wu, name: Songcen Xu, name: Wei Zhang, name: Yu-Gang Jiang, name: Hang Xu
Published: 2023-09-07 08:12:58

Inspired by the remarkable success of Latent Diffusion Models (LDMs) for
image synthesis, we study LDM for text-to-video generation, which is a
formidable challenge due to the computational and memory constraints during
both model training and inference. A single LDM is usually only capable of
generating a very limited number of video frames. Some existing works focus on
separate prediction models for generating more video frames, which suffer from
additional training cost and frame-level jittering, however. In this paper, we
propose a framework called "Reuse and Diffuse" dubbed VidRD to
produce more frames following the frames already generated by an LDM.
Conditioned on an initial video clip with a small number of frames, additional
frames are iteratively generated by reusing the original latent features and
following the previous diffusion process. Besides, for the autoencoder used for
translation between pixel space and latent space, we inject temporal layers
into its decoder and fine-tune these layers for higher temporal consistency. We
also propose a set of strategies for composing video-text data that involve
diverse content from multiple existing datasets including video datasets for
action recognition and image-text datasets. Extensive experiments show that our
method achieves good results in both quantitative and qualitative evaluations.
Our project page is available
https://anonymous0x233.github.io/ReuseAndDiffuse/{here}.
Text2Control3D: Controllable 3D Avatar Generation in Neural Radiance
  Fields using Geometry-Guided Text-to-Image Diffusion Model
Authors: name: Sungwon Hwang, name: Junha Hyung, name: Jaegul Choo
Published: 2023-09-07 08:14:46

Recent advances in diffusion models such as ControlNet have enabled
geometrically controllable, high-fidelity text-to-image generation. However,
none of them addresses the question of adding such controllability to
text-to-3D generation. In response, we propose Text2Control3D, a controllable
text-to-3D avatar generation method whose facial expression is controllable
given a monocular video casually captured with hand-held camera. Our main
strategy is to construct the 3D avatar in Neural Radiance Fields (NeRF)
optimized with a set of controlled viewpoint-aware images that we generate from
ControlNet, whose condition input is the depth map extracted from the input
video. When generating the viewpoint-aware images, we utilize cross-reference
attention to inject well-controlled, referential facial expression and
appearance via cross attention. We also conduct low-pass filtering of Gaussian
latent of the diffusion model in order to ameliorate the viewpoint-agnostic
texture problem we observed from our empirical analysis, where the
viewpoint-aware images contain identical textures on identical pixel positions
that are incomprehensible in 3D. Finally, to train NeRF with the images that
are viewpoint-aware yet are not strictly consistent in geometry, our approach
considers per-image geometric variation as a view of deformation from a shared
3D canonical space. Consequently, we construct the 3D avatar in a canonical
space of deformable NeRF by learning a set of per-image deformation via
deformation field table. We demonstrate the empirical results and discuss the
effectiveness of our method.
ImageBind-LLM: Multi-modality Instruction Tuning
Authors: name: Jiaming Han, name: Renrui Zhang, name: Wenqi Shao, name: Peng Gao, name: Peng Xu, name: Han Xiao, name: Kaipeng Zhang, name: Chris Liu, name: Song Wen, name: Ziyu Guo, name: Xudong Lu, name: Shuai Ren, name: Yafei Wen, name: Xiaoxin Chen, name: Xiangyu Yue, name: Hongsheng Li, name: Yu Qiao
Published: 2023-09-07 17:59:45

We present ImageBind-LLM, a multi-modality instruction tuning method of large
language models (LLMs) via ImageBind. Existing works mainly focus on language
and image instruction tuning, different from which, our ImageBind-LLM can
respond to multi-modality conditions, including audio, 3D point clouds, video,
and their embedding-space arithmetic by only image-text alignment training.
During training, we adopt a learnable bind network to align the embedding space
between LLaMA and ImageBind's image encoder. Then, the image features
transformed by the bind network are added to word tokens of all layers in
LLaMA, which progressively injects visual instructions via an attention-free
and zero-initialized gating mechanism. Aided by the joint embedding of
ImageBind, the simple image-text training enables our model to exhibit superior
multi-modality instruction-following capabilities. During inference, the
multi-modality inputs are fed into the corresponding ImageBind encoders, and
processed by a proposed visual cache model for further cross-modal embedding
enhancement. The training-free cache model retrieves from three million image
features extracted by ImageBind, which effectively mitigates the
training-inference modality discrepancy. Notably, with our approach,
ImageBind-LLM can respond to instructions of diverse modalities and demonstrate
significant language generation quality. Code is released at
https://github.com/OpenGVLab/LLaMA-Adapter.
DoLa: Decoding by Contrasting Layers Improves Factuality in Large
  Language Models
Authors: name: Yung-Sung Chuang, name: Yujia Xie, name: Hongyin Luo, name: Yoon Kim, name: James Glass, name: Pengcheng He
Published: 2023-09-07 17:45:31

Despite their impressive capabilities, large language models (LLMs) are prone
to hallucinations, i.e., generating content that deviates from facts seen
during pretraining. We propose a simple decoding strategy for reducing
hallucinations with pretrained LLMs that does not require conditioning on
retrieved external knowledge nor additional fine-tuning. Our approach obtains
the next-token distribution by contrasting the differences in logits obtained
from projecting the later layers versus earlier layers to the vocabulary space,
exploiting the fact that factual knowledge in an LLMs has generally been shown
to be localized to particular transformer layers. We find that this Decoding by
Contrasting Layers (DoLa) approach is able to better surface factual knowledge
and reduce the generation of incorrect facts. DoLa consistently improves the
truthfulness across multiple choices tasks and open-ended generation tasks, for
example improving the performance of LLaMA family models on TruthfulQA by
12-17% absolute points, demonstrating its potential in making LLMs reliably
generate truthful facts.
FLM-101B: An Open LLM and How to Train It with $100K Budget
Authors: name: Xiang Li, name: Yiqun Yao, name: Xin Jiang, name: Xuezhi Fang, name: Xuying Meng, name: Siqi Fan, name: Peng Han, name: Jing Li, name: Li Du, name: Bowen Qin, name: Zheng Zhang, name: Aixin Sun, name: Yequan Wang
Published: 2023-09-07 17:07:36

Large language models (LLMs) have achieved remarkable success in NLP and
multimodal tasks. Despite these successes, their development faces two main
challenges: (i) high computational cost; and (ii) difficulty in conducting fair
and objective evaluations. LLMs are prohibitively expensive, making it feasible
for only a few major players to undertake their training, thereby constraining
both research and application opportunities. This underscores the importance of
cost-effective LLM training. In this paper, we utilize a growth strategy to
significantly reduce LLM training cost. We demonstrate that an LLM with 101B
parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
existing evaluations that focus more on knowledge-oriented abilities. We
introduce our benchmark including evaluations on important aspects of
intelligence including symbolic mapping, itrule understanding, pattern mining,
and anti-interference. Such evaluations minimize the potential impact of
memorization. Experimental results show that our model FLM-101B, trained with a
budget of 100K, achieves comparable performance to powerful and well-known
models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
contexts unseen in training data. The checkpoint of FLM-101B will be
open-sourced at https://huggingface.co/CofeAI/FLM-101B.
XGen-7B Technical Report
Authors: name: Erik Nijkamp, name: Tian Xie, name: Hiroaki Hayashi, name: Bo Pang, name: Congying Xia, name: Chen Xing, name: Jesse Vig, name: Semih Yavuz, name: Philippe Laban, name: Ben Krause, name: Senthil Purushwalkam, name: Tong Niu, name: Wojciech Kryściński, name: Lidiya Murakhovs'ka, name: Prafulla Kumar Choubey, name: Alex Fabbri, name: Ye Liu, name: Rui Meng, name: Lifu Tu, name: Meghana Bhat, name: Chien-Sheng Wu, name: Silvio Savarese, name: Yingbo Zhou, name: Shafiq Joty, name: Caiming Xiong
Published: 2023-09-07 02:20:03

Large Language Models (LLMs) have become ubiquitous across various domains,
transforming the way we interact with information and conduct research.
However, most high-performing LLMs remain confined behind proprietary walls,
hindering scientific progress. Most open-source LLMs, on the other hand, are
limited in their ability to support longer sequence lengths, which is a key
requirement for many tasks that require inference over an input context. To
address this, we have trained XGen, a series of 7B parameter models on up to 8K
sequence length for up to 1.5T tokens. We have also finetuned the XGen models
on public-domain instructional data, creating their instruction-tuned
counterparts (XGen-Inst). We open-source our models for both research
advancements and commercial applications. Our evaluation on standard benchmarks
shows that XGen models achieve comparable or better results when compared with
state-of-the-art open-source LLMs. Our targeted evaluation on long sequence
modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence
open-source LLMs.
Large Language Models as Optimizers
Authors: name: Chengrun Yang, name: Xuezhi Wang, name: Yifeng Lu, name: Hanxiao Liu, name: Quoc V. Le, name: Denny Zhou, name: Xinyun Chen
Published: 2023-09-07 00:07:15

Optimization is ubiquitous. While derivative-based algorithms have been
powerful tools for various problems, the absence of gradient imposes challenges
on many real-world applications. In this work, we propose Optimization by
PROmpting (OPRO), a simple and effective approach to leverage large language
models (LLMs) as optimizers, where the optimization task is described in
natural language. In each optimization step, the LLM generates new solutions
from the prompt that contains previously generated solutions with their values,
then the new solutions are evaluated and added to the prompt for the next
optimization step. We first showcase OPRO on linear regression and traveling
salesman problems, then move on to prompt optimization where the goal is to
find instructions that maximize the task accuracy. With a variety of LLMs, we
demonstrate that the best prompts optimized by OPRO outperform human-designed
prompts by up to 8% on GSM8K, and by up to 50% on Big-Bench Hard tasks.
GPT Can Solve Mathematical Problems Without a Calculator
Authors: name: Zhen Yang, name: Ming Ding, name: Qingsong Lv, name: Zhihuan Jiang, name: Zehai He, name: Yuyi Guo, name: Jinfeng Bai, name: Jie Tang
Published: 2023-09-06 06:18:16

Previous studies have typically assumed that large language models are unable
to accurately perform arithmetic operations, particularly multiplication of >8
digits, and operations involving decimals and fractions, without the use of
calculator tools. This paper aims to challenge this misconception. With
sufficient training data, a 2 billion-parameter language model can accurately
perform multi-digit arithmetic operations with almost 100% accuracy without
data leakage, significantly surpassing GPT-4 (whose multi-digit multiplication
accuracy is only 4.3%). We also demonstrate that our MathGLM, fine-tuned from
GLM-10B on a dataset with additional multi-step arithmetic operations and math
problems described in text, achieves similar performance to GPT-4 on a
5,000-samples Chinese math problem test set.
Matcha-TTS: A fast TTS architecture with conditional flow matching
Authors: name: Shivam Mehta, name: Ruibo Tu, name: Jonas Beskow, name: Éva Székely, name: Gustav Eje Henter
Published: 2023-09-06 17:59:57

We introduce Matcha-TTS, a new encoder-decoder architecture for speedy TTS
acoustic modelling, trained using optimal-transport conditional flow matching
(OT-CFM). This yields an ODE-based decoder capable of high output quality in
fewer synthesis steps than models trained using score matching. Careful design
choices additionally ensure each synthesis step is fast to run. The method is
probabilistic, non-autoregressive, and learns to speak from scratch without
external alignments. Compared to strong pre-trained baseline models, the
Matcha-TTS system has the smallest memory footprint, rivals the speed of the
fastest models on long utterances, and attains the highest mean opinion score
in a listening test. Please see https://shivammehta25.github.io/Matcha-TTS/ for
audio examples, code, and pre-trained models.