Orionnnn/hub-stats huggingface dataset:
T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional
  Text-to-image Generation
Authors: name: Kaiyi Huang, name: Kaiyue Sun, name: Enze Xie, name: Zhenguo Li, name: Xihui Liu
Published: 2023-07-12 17:59:42

Despite the stunning ability to generate high-quality images by recent
text-to-image models, current approaches often struggle to effectively compose
objects with different attributes and relationships into a complex and coherent
scene. We propose T2I-CompBench, a comprehensive benchmark for open-world
compositional text-to-image generation, consisting of 6,000 compositional text
prompts from 3 categories (attribute binding, object relationships, and complex
compositions) and 6 sub-categories (color binding, shape binding, texture
binding, spatial relationships, non-spatial relationships, and complex
compositions). We further propose several evaluation metrics specifically
designed to evaluate compositional text-to-image generation. We introduce a new
approach, Generative mOdel fine-tuning with Reward-driven Sample selection
(GORS), to boost the compositional text-to-image generation abilities of
pretrained text-to-image models. Extensive experiments and evaluations are
conducted to benchmark previous methods on T2I-CompBench, and to validate the
effectiveness of our proposed evaluation metrics and GORS approach. Project
page is available at https://karine-h.github.io/T2I-CompBench/.
In-context Autoencoder for Context Compression in a Large Language Model
Authors: name: Tao Ge, name: Jing Hu, name: Xun Wang, name: Si-Qing Chen, name: Furu Wei
Published: 2023-07-13 17:59:21

We propose the In-context Autoencoder (ICAE) for context compression in a
large language model (LLM). The ICAE has two modules: a learnable encoder
adapted with LoRA from an LLM for compressing a long context into a limited
number of memory slots, and a fixed decoder which is the target LLM that can
condition on the memory slots for various purposes. We first pretrain the ICAE
using both autoencoding and language modeling objectives on massive text data,
enabling it to generate memory slots that accurately and comprehensively
represent the original context. Then, we fine-tune the pretrained ICAE on a
small amount of instruct data to enhance its interaction with various prompts
for producing desirable responses. Our experimental results demonstrate that
the ICAE learned with our proposed pretraining and fine-tuning paradigm can
effectively produce memory slots with 4times context compression, which can
be well conditioned on by the target LLM to respond to various prompts. The
promising results demonstrate significant implications of the ICAE for its
novel approach to the long context problem and its potential to reduce
computation and memory overheads for LLM inference in practice, suggesting
further research effort in context management for an LLM. Our code and data
will be released shortly.
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding
  and Generation
Authors: name: Yi Wang, name: Yinan He, name: Yizhuo Li, name: Kunchang Li, name: Jiashuo Yu, name: Xin Ma, name: Xinyuan Chen, name: Yaohui Wang, name: Ping Luo, name: Ziwei Liu, name: Yali Wang, name: Limin Wang, name: Yu Qiao
Published: 2023-07-13 17:58:32

This paper introduces InternVid, a large-scale video-centric multimodal
dataset that enables learning powerful and transferable video-text
representations for multimodal understanding and generation. The InternVid
dataset contains over 7 million videos lasting nearly 760K hours, yielding 234M
video clips accompanied by detailed descriptions of total 4.1B words. Our core
contribution is to develop a scalable approach to autonomously build a
high-quality video-text dataset with large language models (LLM), thereby
showcasing its efficacy in learning video-language representation at scale.
Specifically, we utilize a multi-scale approach to generate video-related
descriptions. Furthermore, we introduce ViCLIP, a video-text representation
learning model based on ViT-L. Learned on InternVid via contrastive learning,
this model demonstrates leading zero-shot action recognition and competitive
video retrieval performance. Beyond basic video understanding tasks like
recognition and retrieval, our dataset and model have broad applications. They
are particularly beneficial for generating interleaved video-text data for
learning a video-centric dialogue system, advancing video-to-text and
text-to-video generation research. These proposed resources provide a tool for
researchers and practitioners interested in multimodal video understanding and
generation.
Generating Benchmarks for Factuality Evaluation of Language Models
Authors: name: Dor Muhlgay, name: Ori Ram, name: Inbal Magar, name: Yoav Levine, name: Nir Ratner, name: Yonatan Belinkov, name: Omri Abend, name: Kevin Leyton-Brown, name: Amnon Shashua, name: Yoav Shoham
Published: 2023-07-13 17:14:38

Before deploying a language model (LM) within a given domain, it is important
to measure its tendency to generate factually incorrect information in that
domain. Existing factual generation evaluation methods focus on facts sampled
from the LM itself, and thus do not control the set of evaluated facts and
might under-represent rare and unlikely facts. We propose FACTOR: Factual
Assessment via Corpus TransfORmation, a scalable approach for evaluating LM
factuality. FACTOR automatically transforms a factual corpus of interest into a
benchmark evaluating an LM's propensity to generate true facts from the corpus
vs. similar but incorrect statements. We use our framework to create two
benchmarks: Wiki-FACTOR and News-FACTOR. We show that: (i) our benchmark scores
increase with model size and improve when the LM is augmented with retrieval;
(ii) benchmark score correlates with perplexity, but the two metrics do not
always agree on model ranking; and (iii) when perplexity and benchmark score
disagree, the latter better reflects factuality in open-ended generation, as
measured by human annotators. We make our data and code publicly available in
https://github.com/AI21Labs/factor.
Distilling Large Language Models for Biomedical Knowledge Extraction: A
  Case Study on Adverse Drug Events
Authors: name: Yu Gu, name: Sheng Zhang, name: Naoto Usuyama, name: Yonas Woldesenbet, name: Cliff Wong, name: Praneeth Sanapathi, name: Mu Wei, name: Naveen Valluri, name: Erika Strandberg, name: Tristan Naumann, name: Hoifung Poon
Published: 2023-07-12 20:08:48

Large language models (LLMs), such as GPT-4, have demonstrated remarkable
capabilities across a wide range of tasks, including health applications. In
this paper, we study how LLMs can be used to scale biomedical knowledge
curation. We find that while LLMs already possess decent competency in
structuring biomedical text, by distillation into a task-specific student model
through self-supervised learning, substantial gains can be attained over
out-of-box LLMs, with additional advantages such as cost, efficiency, and
white-box model access.
  We conduct a case study on adverse drug event (ADE) extraction, which is an
important area for improving care. On standard ADE extraction evaluation, a
GPT-3.5 distilled PubMedBERT model attained comparable accuracy as supervised
state-of-the-art models without using any labeled data. Despite being over
1,000 times smaller, the distilled model outperformed its teacher GPT-3.5 by
over 6 absolute points in F1 and GPT-4 by over 5 absolute points.
  Ablation studies on distillation model choice (e.g., PubMedBERT vs BioGPT)
and ADE extraction architecture shed light on best practice for biomedical
knowledge extraction. Similar gains were attained by distillation for other
standard biomedical knowledge extraction tasks such as gene-disease
associations and protected health information, further illustrating the promise
of this approach.
DNAGPT: A Generalized Pre-trained Tool for Versatile DNA Sequence
  Analysis Tasks
Authors: name: Daoan Zhang, name: Weitong Zhang, name: Bing He, name: Jianguo Zhang, name: Chenchen Qin, name: Jianhua Yao
Published: 2023-07-11 06:30:43

Pre-trained large language models demonstrate potential in extracting
information from DNA sequences, yet adapting to a variety of tasks and data
modalities remains a challenge. To address this, we propose DNAGPT, a
generalized DNA pre-training model trained on over 200 billion base pairs from
all mammals. By enhancing the classic GPT model with a binary classification
task (DNA sequence order), a numerical regression task (guanine-cytosine
content prediction), and a comprehensive token language, DNAGPT can handle
versatile DNA analysis tasks while processing both sequence and numerical data.
Our evaluation of genomic signal and region recognition, mRNA abundance
regression, and artificial genomes generation tasks demonstrates DNAGPT's
superior performance compared to existing models designed for specific
downstream tasks, benefiting from pre-training using the newly designed model
structure.
Instruction Mining: High-Quality Instruction Data Selection for Large
  Language Models
Authors: name: Yihan Cao, name: Yanbin Kang, name: Lichao Sun
Published: 2023-07-12 16:37:31

Large language models typically undergo two training stages, pretraining and
finetuning. Despite that large-scale pretraining endows the model with strong
capabilities to generate natural language responses, these pretrained models
can still fail to understand human instructions at times. To enhance language
models' ability of interpreting and responding to instructions, instruction
finetuning has emerged as a critical method in this area. Recent studies found
that large language models can be finetuned to perform well even with a small
amount of high-quality instruction-following data. However, the selection of
high-quality datasets for finetuning language models still lacks clear
guidelines to follow. In this paper, we propose InstructMining, a linear rule
for evaluating instruction-following data quality. We formulate InstructMining
using specific natural language indicators. To investigate the relationship
between data quality and these indicators, we further conduct extensive
finetuning experiments. The experiment results are then applied to estimating
parameters in InstructMining. To further investigate its performance, we use
InstructMining to select high-quality data from unseen datasets. Results
demonstrate that InstructMining can help select relatively high-quality samples
from various instruction-following datasets. Compared to models finetuned on
unfiltered datasets, models finetuned on InstructMining selected datasets
perform better on 42.5% cases.
Giving Robots a Hand: Learning Generalizable Manipulation with
  Eye-in-Hand Human Video Demonstrations
Authors: name: Moo Jin Kim, name: Jiajun Wu, name: Chelsea Finn
Published: 2023-07-12 07:04:53

Eye-in-hand cameras have shown promise in enabling greater sample efficiency
and generalization in vision-based robotic manipulation. However, for robotic
imitation, it is still expensive to have a human teleoperator collect large
amounts of expert demonstrations with a real robot. Videos of humans performing
tasks, on the other hand, are much cheaper to collect since they eliminate the
need for expertise in robotic teleoperation and can be quickly captured in a
wide range of scenarios. Therefore, human video demonstrations are a promising
data source for learning generalizable robotic manipulation policies at scale.
In this work, we augment narrow robotic imitation datasets with broad unlabeled
human video demonstrations to greatly enhance the generalization of eye-in-hand
visuomotor policies. Although a clear visual domain gap exists between human
and robot data, our framework does not need to employ any explicit domain
adaptation method, as we leverage the partial observability of eye-in-hand
cameras as well as a simple fixed image masking scheme. On a suite of eight
real-world tasks involving both 3-DoF and 6-DoF robot arm control, our method
improves the success rates of eye-in-hand manipulation policies by 58%
(absolute) on average, enabling robots to generalize to both new environment
configurations and new tasks that are unseen in the robot demonstration data.
See video results at https://giving-robots-a-hand.github.io/ .
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
  Resolution
Authors: name: Mostafa Dehghani, name: Basil Mustafa, name: Josip Djolonga, name: Jonathan Heek, name: Matthias Minderer, name: Mathilde Caron, name: Andreas Steiner, name: Joan Puigcerver, name: Robert Geirhos, name: Ibrahim Alabdulmohsin, name: Avital Oliver, name: Piotr Padlewski, name: Alexey Gritsenko, name: Mario Lučić, name: Neil Houlsby
Published: 2023-07-12 17:01:03

The ubiquitous and demonstrably suboptimal choice of resizing images to a
fixed resolution before processing them with computer vision models has not yet
been successfully challenged. However, models such as the Vision Transformer
(ViT) offer flexible sequence-based modeling, and hence varying input sequence
lengths. We take advantage of this with NaViT (Native Resolution ViT) which
uses sequence packing during training to process inputs of arbitrary
resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
improved training efficiency for large-scale supervised and contrastive
image-text pretraining. NaViT can be efficiently transferred to standard tasks
such as image and video classification, object detection, and semantic
segmentation and leads to improved results on robustness and fairness
benchmarks. At inference time, the input resolution flexibility can be used to
smoothly navigate the test-time cost-performance trade-off. We believe that
NaViT marks a departure from the standard, CNN-designed, input and modelling
pipeline used by most computer vision models, and represents a promising
direction for ViTs.
SayPlan: Grounding Large Language Models using 3D Scene Graphs for
  Scalable Task Planning
Authors: name: Krishan Rana, name: Jesse Haviland, name: Sourav Garg, name: Jad Abou-Chakra, name: Ian Reid, name: Niko Suenderhauf
Published: 2023-07-12 12:37:55

Large language models (LLMs) have demonstrated impressive results in
developing generalist planning agents for diverse tasks. However, grounding
these plans in expansive, multi-floor, and multi-room environments presents a
significant challenge for robotics. We introduce SayPlan, a scalable approach
to LLM-based, large-scale task planning for robotics using 3D scene graph
(3DSG) representations. To ensure the scalability of our approach, we: (1)
exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic
search for task-relevant subgraphs from a smaller, collapsed representation of
the full graph; (2) reduce the planning horizon for the LLM by integrating a
classical path planner and (3) introduce an iterative replanning pipeline that
refines the initial plan using feedback from a scene graph simulator,
correcting infeasible actions and avoiding planning failures. We evaluate our
approach on two large-scale environments spanning up to 3 floors, 36 rooms and
140 objects, and show that our approach is capable of grounding large-scale,
long-horizon task plans from abstract, and natural language instruction for a
mobile manipulator robot to execute.
PolyLM: An Open Source Polyglot Large Language Model
Authors: name: Xiangpeng Wei, name: Haoran Wei, name: Huan Lin, name: Tianhao Li, name: Pei Zhang, name: Xingzhang Ren, name: Mei Li, name: Yu Wan, name: Zhiwei Cao, name: Binbin Xie, name: Tianxiang Hu, name: Shangjie Li, name: Binyuan Hui, name: Bowen Yu, name: Dayiheng Liu, name: Baosong Yang, name: Fei Huang, name: Jun Xie
Published: 2023-07-12 09:00:37

Large language models (LLMs) demonstrate remarkable ability to comprehend,
reason, and generate following nature language instructions. However, the
development of LLMs has been primarily focused on high-resource languages, such
as English, thereby limiting their applicability and research in other
languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
multilingual capabilities, we 1) integrate bilingual data into training data;
and 2) adopt a curriculum learning strategy that increases the proportion of
non-English data from 30% in the first stage to 60% in the final stage during
pre-training. Further, we propose a multilingual self-instruct method which
automatically generates 132.7K diverse multilingual instructions for model
fine-tuning. To assess the model's performance, we collect several existing
multilingual tasks, including multilingual understanding, question answering,
generation, and translation. Extensive experiments show that PolyLM surpasses
other open-source models such as LLaMA and BLOOM on multilingual tasks while
maintaining comparable performance in English. Our models, alone with the
instruction data and multilingual benchmark, are available at:
https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.
VoxPoser: Composable 3D Value Maps for Robotic Manipulation with
  Language Models
Authors: name: Wenlong Huang, name: Chen Wang, name: Ruohan Zhang, name: Yunzhu Li, name: Jiajun Wu, name: Li Fei-Fei
Published: 2023-07-12 07:40:48

Large language models (LLMs) are shown to possess a wealth of actionable
knowledge that can be extracted for robot manipulation in the form of reasoning
and planning. Despite the progress, most still rely on pre-defined motion
primitives to carry out the physical interactions with the environment, which
remains a major bottleneck. In this work, we aim to synthesize robot
trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
large variety of manipulation tasks given an open-set of instructions and an
open-set of objects. We achieve this by first observing that LLMs excel at
inferring affordances and constraints given a free-form language instruction.
More importantly, by leveraging their code-writing capabilities, they can
interact with a visual-language model (VLM) to compose 3D value maps to ground
the knowledge into the observation space of the agent. The composed value maps
are then used in a model-based planning framework to zero-shot synthesize
closed-loop robot trajectories with robustness to dynamic perturbations. We
further demonstrate how the proposed framework can benefit from online
experiences by efficiently learning a dynamics model for scenes that involve
contact-rich interactions. We present a large-scale study of the proposed
method in both simulated and real-robot environments, showcasing the ability to
perform a large variety of everyday manipulation tasks specified in free-form
natural language. Project website: https://voxposer.github.io
Towards Robust and Efficient Continual Language Learning
Authors: name: Adam Fisch, name: Amal Rannen-Triki, name: Razvan Pascanu, name: Jörg Bornschein, name: Angeliki Lazaridou, name: Elena Gribovskaya, name: Marc'Aurelio Ranzato
Published: 2023-07-11 19:08:31

As the application space of language models continues to evolve, a natural
question to ask is how we can quickly adapt models to new tasks. We approach
this classic question from a continual learning perspective, in which we aim to
continue fine-tuning models trained on past tasks on new tasks, with the goal
of "transferring" relevant knowledge. However, this strategy also runs the risk
of doing more harm than good, i.e., negative transfer. In this paper, we
construct a new benchmark of task sequences that target different possible
transfer scenarios one might face, such as a sequence of tasks with high
potential of positive transfer, high potential for negative transfer, no
expected effect, or a mixture of each. An ideal learner should be able to
maximally exploit information from all tasks that have any potential for
positive transfer, while also avoiding the negative effects of any distracting
tasks that may confuse it. We then propose a simple, yet effective, learner
that satisfies many of our desiderata simply by leveraging a selective strategy
for initializing new models from past task checkpoints. Still, limitations
remain, and we hope this benchmark can help the community to further build and
analyze such learners.
Stack More Layers Differently: High-Rank Training Through Low-Rank
  Updates
Authors: name: Vladislav Lialin, name: Namrata Shivagunde, name: Sherin Muckatira, name: Anna Rumshisky
Published: 2023-07-11 18:02:09

Despite the dominance and effectiveness of scaling, resulting in large
networks with hundreds of billions of parameters, the necessity to train
overparametrized models remains poorly understood, and alternative approaches
do not necessarily make it cheaper to train high-performance models. In this
paper, we explore low-rank training techniques as an alternative approach to
training large neural networks. We introduce a novel method called ReLoRA,
which utilizes low-rank updates to train high-rank networks. We apply ReLoRA to
pre-training transformer language models with up to 350M parameters and
demonstrate comparable performance to regular neural network training.
Furthermore, we observe that the efficiency of ReLoRA increases with model
size, making it a promising approach for training multi-billion-parameter
networks efficiently. Our findings shed light on the potential of low-rank
training techniques and their implications for scaling laws.
SITTA: A Semantic Image-Text Alignment for Image Captioning
Authors: name: Fabian Paischer, name: Thomas Adler, name: Markus Hofmarcher, name: Sepp Hochreiter
Published: 2023-07-10 17:59:21

Textual and semantic comprehension of images is essential for generating
proper captions. The comprehension requires detection of objects, modeling of
relations between them, an assessment of the semantics of the scene and,
finally, representing the extracted knowledge in a language space. To achieve
rich language capabilities while ensuring good image-language mappings,
pretrained language models (LMs) were conditioned on pretrained multi-modal
(image-text) models that allow for image inputs. This requires an alignment of
the image representation of the multi-modal model with the language
representations of a generative LM. However, it is not clear how to best
transfer semantics detected by the vision encoder of the multi-modal model to
the LM. We introduce two novel ways of constructing a linear mapping that
successfully transfers semantics between the embedding spaces of the two
pretrained models. The first aligns the embedding space of the multi-modal
language encoder with the embedding space of the pretrained LM via token
correspondences. The latter leverages additional data that consists of
image-text pairs to construct the mapping directly from vision to language
space. Using our semantic mappings, we unlock image captioning for LMs without
access to gradient information. By using different sources of data we achieve
strong captioning performance on MS-COCO and Flickr30k datasets. Even in the
face of limited data, our method partly exceeds the performance of other
zero-shot and even finetuned competitors. Our ablation studies show that even
LMs at a scale of merely 250M parameters can generate decent captions employing
our semantic mappings. Our approach makes image captioning more accessible for
institutions with restricted computational resources.