Orionnnn/hub-stats huggingface dataset:
FormNetV2: Multimodal Graph Contrastive Learning for Form Document
  Information Extraction
Authors: name: Chen-Yu Lee, name: Chun-Liang Li, name: Hao Zhang, name: Timothy Dozat, name: Vincent Perot, name: Guolong Su, name: Xiang Zhang, name: Kihyuk Sohn, name: Nikolai Glushnev, name: Renshen Wang, name: Joshua Ainslie, name: Shangbang Long, name: Siyang Qin, name: Yasuhisa Fujii, name: Nan Hua, name: Tomas Pfister
Published: 2023-05-04 05:02:04

The recent advent of self-supervised pre-training techniques has led to a
surge in the use of multimodal learning in form document understanding.
However, existing approaches that extend the mask language modeling to other
modalities require careful multi-task tuning, complex reconstruction target
designs, or additional pre-training data. In FormNetV2, we introduce a
centralized multimodal graph contrastive learning strategy to unify
self-supervised pre-training for all modalities in one loss. The graph
contrastive objective maximizes the agreement of multimodal representations,
providing a natural interplay for all modalities without special customization.
In addition, we extract image features within the bounding box that joins a
pair of tokens connected by a graph edge, capturing more targeted visual cues
without loading a sophisticated and separately pre-trained image embedder.
FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
and Payment benchmarks with a more compact model size.
AutoML-GPT: Automatic Machine Learning with GPT
Authors: name: Shujian Zhang, name: Chengyue Gong, name: Lemeng Wu, name: Xingchao Liu, name: Mingyuan Zhou
Published: 2023-05-04 02:09:43

AI tasks encompass a wide range of domains and fields. While numerous AI
models have been designed for specific tasks and applications, they often
require considerable human efforts in finding the right model architecture,
optimization algorithm, and hyperparameters. Recent advances in large language
models (LLMs) like ChatGPT show remarkable capabilities in various aspects of
reasoning, comprehension, and interaction. Consequently, we propose developing
task-oriented prompts and automatically utilizing LLMs to automate the training
pipeline. To implement this concept, we present the AutoML-GPT, which employs
GPT as the bridge to diverse AI models and dynamically trains models with
optimized hyperparameters. AutoML-GPT dynamically takes user requests from the
model and data cards and composes the corresponding prompt paragraph.
Ultimately, with this prompt paragraph, AutoML-GPT will automatically conduct
the experiments from data processing to model architecture, hyperparameter
tuning, and predicted training log. By leveraging {\ours}'s robust language
capabilities and the available AI models, AutoML-GPT can tackle numerous
intricate AI tasks across various tasks and datasets. This approach achieves
remarkable results in computer vision, natural language processing, and other
challenging areas. Extensive experiments and ablation studies demonstrate that
our method can be general, effective, and beneficial for many AI tasks.
Cheaply Evaluating Inference Efficiency Metrics for Autoregressive
  Transformer APIs
Authors: name: Deepak Narayanan, name: Keshav Santhanam, name: Peter Henderson, name: Rishi Bommasani, name: Tony Lee, name: Percy Liang
Published: 2023-05-03 21:51:42

Large language models (LLMs) power many state-of-the-art systems in natural
language processing. However, these models are extremely computationally
expensive, even at inference time, raising the natural question: when is the
extra cost of deploying a larger model worth the anticipated boost in
capabilities? Better understanding this tradeoff fundamentally could benefit
from an inference efficiency metric that is both (i) easily comparable across
models from different providers, and (ii) representative of the true cost of
running queries in an isolated performance environment. Unfortunately, access
to LLMs today is largely restricted to black-box text generation APIs and raw
runtimes measured through this interface do not satisfy these desiderata: model
providers can apply various software and hardware optimizations orthogonal to
the model, and models served on shared infrastructure are susceptible to
performance contention. To circumvent these problems, we propose a new metric
for comparing inference efficiency across models. This metric puts models on
equal footing as though they were served (i) on uniform hardware and software,
and (ii) without performance contention. We call this metric the
idealized runtime, and we propose a methodology to efficiently estimate
this metric for autoregressive Transformer models. We also propose cost-aware
variants that incorporate the number of accelerators needed to serve the model.
Using these metrics, we compare ten state-of-the-art LLMs to provide the first
analysis of inference efficiency-capability tradeoffs; we make several
observations from this analysis, including the fact that the superior inference
runtime performance of certain APIs is often a byproduct of optimizations
within the API rather than the underlying model. Our methodology also
facilitates the efficient comparison of different software and hardware stacks.
Principle-Driven Self-Alignment of Language Models from Scratch with
  Minimal Human Supervision
Authors: name: Zhiqing Sun, name: Yikang Shen, name: Qinhong Zhou, name: Hongxin Zhang, name: Zhenfang Chen, name: David Cox, name: Yiming Yang, name: Chuang Gan
Published: 2023-05-04 17:59:28

Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised
fine-tuning (SFT) with human annotations and reinforcement learning from human
feedback (RLHF) to align the output of large language models (LLMs) with human
intentions, ensuring they are helpful, ethical, and reliable. However, this
dependence can significantly constrain the true potential of AI-assistant
agents due to the high cost of obtaining human supervision and the related
issues on quality, reliability, diversity, self-consistency, and undesirable
biases. To address these challenges, we propose a novel approach called
SELF-ALIGN, which combines principle-driven reasoning and the generative power
of LLMs for the self-alignment of AI agents with minimal human supervision. Our
approach encompasses four stages: first, we use an LLM to generate synthetic
prompts, and a topic-guided method to augment the prompt diversity; second, we
use a small set of human-written principles for AI models to follow, and guide
the LLM through in-context learning from demonstrations (of principles
application) to produce helpful, ethical, and reliable responses to user's
queries; third, we fine-tune the original LLM with the high-quality
self-aligned responses so that the resulting model can generate desirable
responses for each query directly without the principle set and the
demonstrations anymore; and finally, we offer a refinement step to address the
issues of overly-brief or indirect responses. Applying SELF-ALIGN to the
LLaMA-65b base language model, we develop an AI assistant named Dromedary. With
fewer than 300 lines of human annotations (including < 200 seed prompts, 16
generic principles, and 5 exemplars for in-context learning). Dromedary
significantly surpasses the performance of several state-of-the-art AI systems,
including Text-Davinci-003 and Alpaca, on benchmark datasets with various
settings.
ChatGPT-steered Editing Instructor for Customization of Abstractive
  Summarization
Authors: name: Wen Xiao, name: Yujia Xie, name: Giuseppe Carenini, name: Pengcheng He
Published: 2023-05-04 01:12:35

Tailoring outputs of large language models, such as ChatGPT, to specific user
needs remains a challenge despite their impressive generation quality. In this
paper, we propose a tri-agent generation pipeline consisting of a generator, an
instructor, and an editor to enhance the customization of generated outputs.
The generator produces an initial output, the user-specific instructor
generates editing instructions, and the editor generates a revised output
aligned with user preferences. The inference-only large language model
(ChatGPT) serves as both the generator and the editor, while a smaller model
acts as the user-specific instructor to guide the generation process toward
user needs. The instructor is trained using editor-steered reinforcement
learning, leveraging feedback from the large-scale editor model to optimize
instruction generation. Experimental results on two abstractive summarization
datasets demonstrate the effectiveness of our approach in generating outputs
that better fulfill user expectations.
Personalize Segment Anything Model with One Shot
Authors: name: Renrui Zhang, name: Zhengkai Jiang, name: Ziyu Guo, name: Shilin Yan, name: Junting Pan, name: Hao Dong, name: Peng Gao, name: Hongsheng Li
Published: 2023-05-04 17:59:36

Driven by large-data pre-training, Segment Anything Model (SAM) has been
demonstrated as a powerful and promptable framework, revolutionizing the
segmentation models. Despite the generality, customizing SAM for specific
visual concepts without man-powered prompting is under explored, e.g.,
automatically segmenting your pet dog in different images. In this paper, we
propose a training-free Personalization approach for SAM, termed as PerSAM.
Given only a single image with a reference mask, PerSAM first localizes the
target concept by a location prior, and segments it within other images or
videos via three techniques: target-guided attention, target-semantic
prompting, and cascaded post-refinement. In this way, we effectively adapt SAM
for private use without any training. To further alleviate the mask ambiguity,
we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the
entire SAM, we introduce two learnable weights for multi-scale masks, only
training 2 parameters within 10 seconds for improved performance. To
demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for
personalized evaluation, and test our methods on video object segmentation with
competitive performance. Besides, our approach can also enhance DreamBooth to
personalize Stable Diffusion for text-to-image generation, which discards the
background disturbance for better target appearance learning. Code is released
at https://github.com/ZrrSkywalker/Personalize-SAM
Shap-E: Generating Conditional 3D Implicit Functions
Authors: name: Heewoo Jun, name: Alex Nichol
Published: 2023-05-03 23:59:13

We present Shap-E, a conditional generative model for 3D assets. Unlike
recent work on 3D generative models which produce a single output
representation, Shap-E directly generates the parameters of implicit functions
that can be rendered as both textured meshes and neural radiance fields. We
train Shap-E in two stages: first, we train an encoder that deterministically
maps 3D assets into the parameters of an implicit function; second, we train a
conditional diffusion model on outputs of the encoder. When trained on a large
dataset of paired 3D and text data, our resulting models are capable of
generating complex and diverse 3D assets in a matter of seconds. When compared
to Point-E, an explicit generative model over point clouds, Shap-E converges
faster and reaches comparable or better sample quality despite modeling a
higher-dimensional, multi-representation output space. We release model
weights, inference code, and samples at https://github.com/openai/shap-e.