Orionnnn/hub-stats huggingface dataset:
MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
Authors: name: Weihao Yu, name: Zhengyuan Yang, name: Linjie Li, name: Jianfeng Wang, name: Kevin Lin, name: Zicheng Liu, name: Xinchao Wang, name: Lijuan Wang
Published: 2023-08-04 17:59:47

We propose MM-Vet, an evaluation benchmark that examines large multimodal
models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various
intriguing abilities, such as solving math problems written on the blackboard,
reasoning about events and celebrities in news images, and explaining visual
jokes. Rapid model advancements pose challenges to evaluation benchmark
development. Problems include: (1) How to systematically structure and evaluate
the complicated multimodal tasks; (2) How to design evaluation metrics that
work well across question and answer types; and (3) How to give model insights
beyond a simple performance ranking. To this end, we present MM-Vet, designed
based on the insight that the intriguing ability to solve complicated tasks is
often achieved by a generalist model being able to integrate different core
vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and
examines the 16 integrations of interest derived from the capability
combination. For evaluation metrics, we propose an LLM-based evaluator for
open-ended outputs. The evaluator enables the evaluation across different
question types and answer styles, resulting in a unified scoring metric. We
evaluate representative LMMs on MM-Vet, providing insights into the
capabilities of different LMM system paradigms and models. Code and data are
available at https://github.com/yuweihao/MM-Vet.
Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen
  Convolutional CLIP
Authors: name: Qihang Yu, name: Ju He, name: Xueqing Deng, name: Xiaohui Shen, name: Liang-Chieh Chen
Published: 2023-08-04 17:59:01

Open-vocabulary segmentation is a challenging task requiring segmenting and
recognizing objects from an open set of categories. One way to address this
challenge is to leverage multi-modal models, such as CLIP, to provide image and
text features in a shared embedding space, which bridges the gap between
closed-vocabulary and open-vocabulary recognition. Hence, existing methods
often adopt a two-stage framework to tackle the problem, where the inputs first
go through a mask generator and then through the CLIP model along with the
predicted masks. This process involves extracting features from images multiple
times, which can be ineffective and inefficient. By contrast, we propose to
build everything into a single-stage framework using a shared Frozen
Convolutional CLIP backbone, which not only significantly simplifies the
current two-stage pipeline, but also remarkably yields a better accuracy-cost
trade-off. The proposed FC-CLIP, benefits from the following observations: the
frozen CLIP backbone maintains the ability of open-vocabulary classification
and can also serve as a strong mask generator, and the convolutional CLIP
generalizes well to a larger input resolution than the one used during
contrastive image-text pretraining. When training on COCO panoptic data only
and testing in a zero-shot manner, FC-CLIP achieve 26.8 PQ, 16.8 AP, and 34.1
mIoU on ADE20K, 18.2 PQ, 27.9 mIoU on Mapillary Vistas, 44.0 PQ, 26.8 AP, 56.2
mIoU on Cityscapes, outperforming the prior art by +4.2 PQ, +2.4 AP, +4.2 mIoU
on ADE20K, +4.0 PQ on Mapillary Vistas and +20.1 PQ on Cityscapes,
respectively. Additionally, the training and testing time of FC-CLIP is 7.5x
and 6.6x significantly faster than the same prior art, while using 5.9x fewer
parameters. FC-CLIP also sets a new state-of-the-art performance across various
open-vocabulary semantic segmentation datasets. Code at
https://github.com/bytedance/fc-clip
Getting the Ball Rolling: Learning a Dexterous Policy for a Biomimetic
  Tendon-Driven Hand with Rolling Contact Joints
Authors: name: Yasunori Toshimitsu, name: Benedek Forrai, name: Barnabas Gavin Cangan, name: Ulrich Steger, name: Manuel Knecht, name: Stefan Weirich, name: Robert K. Katzschmann
Published: 2023-08-04 16:39:27

Biomimetic, dexterous robotic hands have the potential to replicate much of
the tasks that a human can do, and to achieve status as a general manipulation
platform. Recent advances in reinforcement learning (RL) frameworks have
achieved remarkable performance in quadrupedal locomotion and dexterous
manipulation tasks. Combined with GPU-based highly parallelized simulations
capable of simulating thousands of robots in parallel, RL-based controllers
have become more scalable and approachable. However, in order to bring
RL-trained policies to the real world, we require training frameworks that
output policies that can work with physical actuators and sensors as well as a
hardware platform that can be manufactured with accessible materials yet is
robust enough to run interactive policies. This work introduces the biomimetic
tendon-driven Faive Hand and its system architecture, which uses tendon-driven
rolling contact joints to achieve a 3D printable, robust high-DoF hand design.
We model each element of the hand and integrate it into a GPU simulation
environment to train a policy with RL, and achieve zero-shot transfer of a
dexterous in-hand sphere rotation skill to the physical robot hand.
Training Data Protection with Compositional Diffusion Models
Authors: name: Aditya Golatkar, name: Alessandro Achille, name: Ashwin Swaminathan, name: Stefano Soatto
Published: 2023-08-02 23:27:49

We introduce Compartmentalized Diffusion Models (CDM), a method to train
different diffusion models (or prompts) on distinct data sources and
arbitrarily compose them at inference time. The individual models can be
trained in isolation, at different times, and on different distributions and
domains and can be later composed to achieve performance comparable to a
paragon model trained on all data simultaneously. Furthermore, each model only
contains information about the subset of the data it was exposed to during
training, enabling several forms of training data protection. In particular,
CDMs are the first method to enable both selective forgetting and continual
learning for large-scale diffusion models, as well as allowing serving
customized models based on the user's access rights. CDMs also allow
determining the importance of a subset of the data in generating particular
samples.
Retroformer: Retrospective Large Language Agents with Policy Gradient
  Optimization
Authors: name: Weiran Yao, name: Shelby Heinecke, name: Juan Carlos Niebles, name: Zhiwei Liu, name: Yihao Feng, name: Le Xue, name: Rithesh Murthy, name: Zeyuan Chen, name: Jianguo Zhang, name: Devansh Arpit, name: Ran Xu, name: Phil Mui, name: Huan Wang, name: Caiming Xiong, name: Silvio Savarese
Published: 2023-08-04 06:14:23

Recent months have seen the emergence of a powerful new trend in which large
language models (LLMs) are augmented to become autonomous language agents
capable of performing objective oriented multi-step tasks on their own, rather
than merely responding to queries from human users. Most existing language
agents, however, are not optimized using environment-specific rewards. Although
some agents enable iterative refinement through verbal feedback, they do not
reason and plan in ways that are compatible with gradient-based learning from
rewards. This paper introduces a principled framework for reinforcing large
language agents by learning a retrospective model, which automatically tunes
the language agent prompts from environment feedback through policy gradient.
Specifically, our proposed agent architecture learns from rewards across
multiple environments and tasks, for fine-tuning a pre-trained language model
which refines the language agent prompt by summarizing the root cause of prior
failed attempts and proposing action plans. Experimental results on various
tasks demonstrate that the language agents improve over time and that our
approach considerably outperforms baselines that do not properly leverage
gradients from the environment. This demonstrates that using policy gradient
optimization to improve language agents, for which we believe our work is one
of the first, seems promising and can be applied to optimize other models in
the agent architecture to enhance agent performances over time.
Scaling Clinical Trial Matching Using Large Language Models: A Case
  Study in Oncology
Authors: name: Cliff Wong, name: Sheng Zheng, name: Yu Gu, name: Christine Moung, name: Jacob Abel, name: Naoto Usuyama, name: Roshanthi Weerasinghe, name: Brian Piening, name: Tristan Naumann, name: Carlo Bifulco, name: Hoifung Poon
Published: 2023-08-04 07:51:15

Clinical trial matching is a key process in health delivery and discovery. In
practice, it is plagued by overwhelming unstructured data and unscalable manual
processing. In this paper, we conduct a systematic study on scaling clinical
trial matching using large language models (LLMs), with oncology as the focus
area. Our study is grounded in a clinical trial matching system currently in
test deployment at a large U.S. health network. Initial findings are promising:
out of box, cutting-edge LLMs, such as GPT-4, can already structure elaborate
eligibility criteria of clinical trials and extract complex matching logic
(e.g., nested AND/OR/NOT). While still far from perfect, LLMs substantially
outperform prior strong baselines and may serve as a preliminary solution to
help triage patient-trial candidates with humans in the loop. Our study also
reveals a few significant growth areas for applying LLMs to end-to-end clinical
trial matching, such as context limitation and accuracy, especially in
structuring patient information from longitudinal medical records.
The All-Seeing Project: Towards Panoptic Visual Recognition and
  Understanding of the Open World
Authors: name: Weiyun Wang, name: Min Shi, name: Qingyun Li, name: Wenhai Wang, name: Zhenhang Huang, name: Linjie Xing, name: Zhe Chen, name: Hao Li, name: Xizhou Zhu, name: Zhiguo Cao, name: Yushi Chen, name: Tong Lu, name: Jifeng Dai, name: Yu Qiao
Published: 2023-08-03 17:59:47

We present the All-Seeing (AS) project: a large-scale data and model for
recognizing and understanding everything in the open world. Using a scalable
data engine that incorporates human feedback and efficient models in the loop,
we create a new dataset (AS-1B) with over 1 billion regions annotated with
semantic tags, question-answering pairs, and detailed captions. It covers a
wide range of 3.5 million common and rare concepts in the real world, and has
132.2 billion tokens that describe the concepts and their attributes.
Leveraging this new dataset, we develop the All-Seeing model (ASM), a unified
framework for panoptic visual recognition and understanding. The model is
trained with open-ended language prompts and locations, which allows it to
generalize to various vision and language tasks with remarkable zero-shot
performance, including region-text retrieval, region recognition, captioning,
and question-answering. We hope that this project can serve as a foundation for
vision-language artificial general intelligence research. Models and the
dataset shall be released at https://github.com/OpenGVLab/All-Seeing, and demo
can be seen at https://huggingface.co/spaces/OpenGVLab/all-seeing.
DETR Doesn't Need Multi-Scale or Locality Design
Authors: name: Yutong Lin, name: Yuhui Yuan, name: Zheng Zhang, name: Chen Li, name: Nanning Zheng, name: Han Hu
Published: 2023-08-03 17:59:04

This paper presents an improved DETR detector that maintains a "plain"
nature: using a single-scale feature map and global cross-attention
calculations without specific locality constraints, in contrast to previous
leading DETR-based detectors that reintroduce architectural inductive biases of
multi-scale and locality into the decoder. We show that two simple technologies
are surprisingly effective within a plain design to compensate for the lack of
multi-scale feature maps and locality constraints. The first is a box-to-pixel
relative position bias (BoxRPB) term added to the cross-attention formulation,
which well guides each query to attend to the corresponding object region while
also providing encoding flexibility. The second is masked image modeling
(MIM)-based backbone pre-training which helps learn representation with
fine-grained localization ability and proves crucial for remedying dependencies
on the multi-scale feature maps. By incorporating these technologies and recent
advancements in training and problem formation, the improved "plain" DETR
showed exceptional improvements over the original DETR detector. By leveraging
the Object365 dataset for pre-training, it achieved 63.9 mAP accuracy using a
Swin-L backbone, which is highly competitive with state-of-the-art detectors
which all heavily rely on multi-scale feature maps and region-based feature
extraction. Code is available at https://github.com/impiga/Plain-DETR .
TDMD: A Database for Dynamic Color Mesh Subjective and Objective Quality
  Explorations
Authors: name: Qi Yang, name: Joel Jung, name: Timon Deschamps, name: Xiaozhong Xu, name: Shan Liu
Published: 2023-08-03 01:50:48

Dynamic colored meshes (DCM) are widely used in various applications;
however, these meshes may undergo different processes, such as compression or
transmission, which can distort them and degrade their quality. To facilitate
the development of objective metrics for DCMs and study the influence of
typical distortions on their perception, we create the Tencent - dynamic
colored mesh database (TDMD) containing eight reference DCM objects with six
typical distortions. Using processed video sequences (PVS) derived from the
DCM, we have conducted a large-scale subjective experiment that resulted in 303
distorted DCM samples with mean opinion scores, making the TDMD the largest
available DCM database to our knowledge. This database enabled us to study the
impact of different types of distortion on human perception and offer
recommendations for DCM compression and related tasks. Additionally, we have
evaluated three types of state-of-the-art objective metrics on the TDMD,
including image-based, point-based, and video-based metrics, on the TDMD. Our
experimental results highlight the strengths and weaknesses of each metric, and
we provide suggestions about the selection of metrics in practical DCM
applications. The TDMD will be made publicly available at the following
location: https://multimedia.tencent.com/resources/tdmd.
HANDAL: A Dataset of Real-World Manipulable Object Categories with Pose
  Annotations, Affordances, and Reconstructions
Authors: name: Andrew Guo, name: Bowen Wen, name: Jianhe Yuan, name: Jonathan Tremblay, name: Stephen Tyree, name: Jeffrey Smith, name: Stan Birchfield
Published: 2023-08-02 23:59:59

We present the HANDAL dataset for category-level object pose estimation and
affordance prediction. Unlike previous datasets, ours is focused on
robotics-ready manipulable objects that are of the proper size and shape for
functional grasping by robot manipulators, such as pliers, utensils, and
screwdrivers. Our annotation process is streamlined, requiring only a single
off-the-shelf camera and semi-automated processing, allowing us to produce
high-quality 3D annotations without crowd-sourcing. The dataset consists of
308k annotated image frames from 2.2k videos of 212 real-world objects in 17
categories. We focus on hardware and kitchen tool objects to facilitate
research in practical scenarios in which a robot manipulator needs to interact
with the environment beyond simple pushing or indiscriminate grasping. We
outline the usefulness of our dataset for 6-DoF category-level pose+scale
estimation and related tasks. We also provide 3D reconstructed meshes of all
objects, and we outline some of the bottlenecks to be addressed for
democratizing the collection of datasets like this one.
Learning to Model the World with Language
Authors: name: Jessy Lin, name: Yuqing Du, name: Olivia Watkins, name: Danijar Hafner, name: Pieter Abbeel, name: Dan Klein, name: Anca Dragan
Published: 2023-07-31 17:57:49

To interact with humans in the world, agents need to understand the diverse
types of language that people use, relate them to the visual world, and act
based on them. While current agents learn to execute simple language
instructions from task rewards, we aim to build agents that leverage diverse
language that conveys general knowledge, describes the state of the world,
provides interactive feedback, and more. Our key idea is that language helps
agents predict the future: what will be observed, how the world will behave,
and which situations will be rewarded. This perspective unifies language
understanding with future prediction as a powerful self-supervised learning
objective. We present Dynalang, an agent that learns a multimodal world model
that predicts future text and image representations and learns to act from
imagined model rollouts. Unlike traditional agents that use language only to
predict actions, Dynalang acquires rich language understanding by using past
language also to predict future language, video, and rewards. In addition to
learning from online interaction in an environment, Dynalang can be pretrained
on datasets of text, video, or both without actions or rewards. From using
language hints in grid worlds to navigating photorealistic scans of homes,
Dynalang utilizes diverse types of language to improve task performance,
including environment descriptions, game rules, and instructions.
Computational Long Exposure Mobile Photography
Authors: name: Eric Tabellion, name: Nikhil Karnad, name: Noa Glaser, name: Ben Weiss, name: David E. Jacobs, name: Yael Pritch
Published: 2023-08-02 18:36:54

Long exposure photography produces stunning imagery, representing moving
elements in a scene with motion-blur. It is generally employed in two
modalities, producing either a foreground or a background blur effect.
Foreground blur images are traditionally captured on a tripod-mounted camera
and portray blurred moving foreground elements, such as silky water or light
trails, over a perfectly sharp background landscape. Background blur images,
also called panning photography, are captured while the camera is tracking a
moving subject, to produce an image of a sharp subject over a background
blurred by relative motion. Both techniques are notoriously challenging and
require additional equipment and advanced skills. In this paper, we describe a
computational burst photography system that operates in a hand-held smartphone
camera app, and achieves these effects fully automatically, at the tap of the
shutter button. Our approach first detects and segments the salient subject. We
track the scene motion over multiple frames and align the images in order to
preserve desired sharpness and to produce aesthetically pleasing motion
streaks. We capture an under-exposed burst and select the subset of input
frames that will produce blur trails of controlled length, regardless of scene
or camera motion velocity. We predict inter-frame motion and synthesize
motion-blur to fill the temporal gaps between the input frames. Finally, we
composite the blurred image with the sharp regular exposure to protect the
sharpness of faces or areas of the scene that are barely moving, and produce a
final high resolution and high dynamic range (HDR) photograph. Our system
democratizes a capability previously reserved to professionals, and makes this
creative style accessible to most casual photographers.
  More information and supplementary material can be found on our project
webpage: https://motion-mode.github.io/
Multimodal Neurons in Pretrained Text-Only Transformers
Authors: name: Sarah Schwettmann, name: Neil Chowdhury, name: Antonio Torralba
Published: 2023-08-03 05:27:12

Language models demonstrate remarkable capacity to generalize representations
learned in one modality to downstream tasks in other modalities. Can we trace
this ability to individual neurons? We study the case where a frozen text
transformer is augmented with vision using a self-supervised visual encoder and
a single linear projection learned on an image-to-text task. Outputs of the
projection layer are not immediately decodable into language describing image
content; instead, we find that translation between modalities occurs deeper
within the transformer. We introduce a procedure for identifying "multimodal
neurons" that convert visual representations into corresponding text, and
decoding the concepts they inject into the model's residual stream. In a series
of experiments, we show that multimodal neurons operate on specific visual
concepts across inputs, and have a systematic causal effect on image
captioning.
MusicLDM: Enhancing Novelty in Text-to-Music Generation Using
  Beat-Synchronous Mixup Strategies
Authors: name: Ke Chen, name: Yusong Wu, name: Haohe Liu, name: Marianna Nezhurina, name: Taylor Berg-Kirkpatrick, name: Shlomo Dubnov
Published: 2023-08-03 05:35:37

Diffusion models have shown promising results in cross-modal generation
tasks, including text-to-image and text-to-audio generation. However,
generating music, as a special type of audio, presents unique challenges due to
limited availability of music data and sensitive issues related to copyright
and plagiarism. In this paper, to tackle these challenges, we first construct a
state-of-the-art text-to-music model, MusicLDM, that adapts Stable Diffusion
and AudioLDM architectures to the music domain. We achieve this by retraining
the contrastive language-audio pretraining model (CLAP) and the Hifi-GAN
vocoder, as components of MusicLDM, on a collection of music data samples.
Then, to address the limitations of training data and to avoid plagiarism, we
leverage a beat tracking model and propose two different mixup strategies for
data augmentation: beat-synchronous audio mixup and beat-synchronous latent
mixup, which recombine training audio directly or via a latent embeddings
space, respectively. Such mixup strategies encourage the model to interpolate
between musical training samples and generate new music within the convex hull
of the training data, making the generated music more diverse while still
staying faithful to the corresponding style. In addition to popular evaluation
metrics, we design several new evaluation metrics based on CLAP score to
demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies
improve both the quality and novelty of generated music, as well as the
correspondence between input text and generated music.
Scaling Relationship on Learning Mathematical Reasoning with Large
  Language Models
Authors: name: Zheng Yuan, name: Hongyi Yuan, name: Chengpeng Li, name: Guanting Dong, name: Chuanqi Tan, name: Chang Zhou
Published: 2023-08-03 15:34:01

Mathematical reasoning is a challenging task for large language models
(LLMs), while the scaling relationship of it with respect to LLM capacity is
under-explored. In this paper, we investigate how the pre-training loss,
supervised data amount, and augmented data amount influence the reasoning
performances of a supervised LLM. We find that pre-training loss is a better
indicator of the model's performance than the model's parameter count. We apply
supervised fine-tuning (SFT) with different amounts of supervised data and
empirically find a log-linear relation between data amount and model
performance, and we find better models improve less with enlarged supervised
datasets. To augment more data samples for improving model performances without
any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT
uses supervised models to generate and collect correct reasoning paths as
augmented fine-tuning datasets. We find with augmented samples containing more
distinct reasoning paths, RFT improves mathematical reasoning performance more
for LLMs. We also find RFT brings more improvement for less performant LLMs.
Furthermore, we combine rejection samples from multiple models which push
LLaMA-7B to an accuracy of 49.3% and outperforms the supervised fine-tuning
(SFT) accuracy of 35.9% significantly.