Orionnnn/hub-stats huggingface dataset:
RestoreFormer++: Towards Real-World Blind Face Restoration from
  Undegraded Key-Value Pairs
Authors: name: Zhouxia Wang, name: Jiawei Zhang, name: Tianshui Chen, name: Wenping Wang, name: Ping Luo
Published: 2023-08-14 16:04:53

Blind face restoration aims at recovering high-quality face images from those
with unknown degradations. Current algorithms mainly introduce priors to
complement high-quality details and achieve impressive progress. However, most
of these algorithms ignore abundant contextual information in the face and its
interplay with the priors, leading to sub-optimal performance. Moreover, they
pay less attention to the gap between the synthetic and real-world scenarios,
limiting the robustness and generalization to real-world applications. In this
work, we propose RestoreFormer++, which on the one hand introduces
fully-spatial attention mechanisms to model the contextual information and the
interplay with the priors, and on the other hand, explores an extending
degrading model to help generate more realistic degraded face images to
alleviate the synthetic-to-real-world gap. Compared with current algorithms,
RestoreFormer++ has several crucial benefits. First, instead of using a
multi-head self-attention mechanism like the traditional visual transformer, we
introduce multi-head cross-attention over multi-scale features to fully explore
spatial interactions between corrupted information and high-quality priors. In
this way, it can facilitate RestoreFormer++ to restore face images with higher
realness and fidelity. Second, in contrast to the recognition-oriented
dictionary, we learn a reconstruction-oriented dictionary as priors, which
contains more diverse high-quality facial details and better accords with the
restoration target. Third, we introduce an extending degrading model that
contains more realistic degraded scenarios for training data synthesizing, and
thus helps to enhance the robustness and generalization of our RestoreFormer++
model. Extensive experiments show that RestoreFormer++ outperforms
state-of-the-art algorithms on both synthetic and real-world datasets.
Jurassic World Remake: Bringing Ancient Fossils Back to Life via
  Zero-Shot Long Image-to-Image Translation
Authors: name: Alexander Martin, name: Haitian Zheng, name: Jie An, name: Jiebo Luo
Published: 2023-08-14 17:59:31

With a strong understanding of the target domain from natural language, we
produce promising results in translating across large domain gaps and bringing
skeletons back to life. In this work, we use text-guided latent diffusion
models for zero-shot image-to-image translation (I2I) across large domain gaps
(longI2I), where large amounts of new visual features and new geometry need to
be generated to enter the target domain. Being able to perform translations
across large domain gaps has a wide variety of real-world applications in
criminology, astrology, environmental conservation, and paleontology. In this
work, we introduce a new task Skull2Animal for translating between skulls and
living animals. On this task, we find that unguided Generative Adversarial
Networks (GANs) are not capable of translating across large domain gaps.
Instead of these traditional I2I methods, we explore the use of guided
diffusion and image editing models and provide a new benchmark model,
Revive-2I, capable of performing zero-shot I2I via text-prompting latent
diffusion models. We find that guidance is necessary for longI2I because, to
bridge the large domain gap, prior knowledge about the target domain is needed.
In addition, we find that prompting provides the best and most scalable
information about the target domain as classifier-guided diffusion models
require retraining for specific use cases and lack stronger constraints on the
target domain because of the wide variety of images they are trained on.
IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image
  Diffusion Models
Authors: name: Hu Ye, name: Jun Zhang, name: Sibo Liu, name: Xiao Han, name: Wei Yang
Published: 2023-08-13 08:34:51

Recent years have witnessed the strong power of large text-to-image diffusion
models for the impressive generative capability to create high-fidelity images.
However, it is very tricky to generate desired images using only text prompt as
it often involves complex prompt engineering. An alternative to text prompt is
image prompt, as the saying goes: "an image is worth a thousand words".
Although existing methods of direct fine-tuning from pretrained models are
effective, they require large computing resources and are not compatible with
other base models, text prompt, and structural controls. In this paper, we
present IP-Adapter, an effective and lightweight adapter to achieve image
prompt capability for the pretrained text-to-image diffusion models. The key
design of our IP-Adapter is decoupled cross-attention mechanism that separates
cross-attention layers for text features and image features. Despite the
simplicity of our method, an IP-Adapter with only 22M parameters can achieve
comparable or even better performance to a fully fine-tuned image prompt model.
As we freeze the pretrained diffusion model, the proposed IP-Adapter can be
generalized not only to other custom models fine-tuned from the same base
model, but also to controllable generation using existing controllable tools.
With the benefit of the decoupled cross-attention strategy, the image prompt
can also work well with the text prompt to achieve multimodal image generation.
The project page is available at https://ip-adapter.github.io.
SpeechX: Neural Codec Language Model as a Versatile Speech Transformer
Authors: name: Xiaofei Wang, name: Manthan Thakker, name: Zhuo Chen, name: Naoyuki Kanda, name: Sefik Emre Eskimez, name: Sanyuan Chen, name: Min Tang, name: Shujie Liu, name: Jinyu Li, name: Takuya Yoshioka
Published: 2023-08-14 01:01:19

Recent advancements in generative speech models based on audio-text prompts
have enabled remarkable innovations like high-quality zero-shot text-to-speech.
However, existing models still face limitations in handling diverse audio-text
speech generation tasks involving transforming input speech and processing
audio captured in adverse acoustic conditions. This paper introduces SpeechX, a
versatile speech generation model capable of zero-shot TTS and various speech
transformation tasks, dealing with both clean and noisy signals. SpeechX
combines neural codec language modeling with multi-task learning using
task-dependent prompting, enabling unified and extensible modeling and
providing a consistent way for leveraging textual input in speech enhancement
and transformation tasks. Experimental results show SpeechX's efficacy in
various tasks, including zero-shot TTS, noise suppression, target speaker
extraction, speech removal, and speech editing with or without background
noise, achieving comparable or superior performance to specialized models
across tasks. See https://aka.ms/speechx for demo samples.
Platypus: Quick, Cheap, and Powerful Refinement of LLMs
Authors: name: Ariel N. Lee, name: Cole J. Hunter, name: Nataniel Ruiz
Published: 2023-08-14 17:59:56

We present Platypus, a family of fine-tuned and merged Large
Language Models (LLMs) that achieves the strongest performance and currently
stands at first place in HuggingFace's Open LLM Leaderboard as of the release
date of this work. In this work we describe (1) our curated dataset
Open-Platypus, that is a subset of other open datasets and which
we release to the public (2) our process of fine-tuning and merging
LoRA modules in order to conserve the strong prior of pretrained LLMs, while
bringing specific domain knowledge to the surface (3) our efforts in checking
for test data leaks and contamination in the training data, which can inform
future research. Specifically, the Platypus family achieves strong performance
in quantitative LLM metrics across model sizes, topping the global Open LLM
leaderboard while using just a fraction of the fine-tuning data and overall
compute that are required for other state-of-the-art fine-tuned LLMs. In
particular, a 13B Platypus model can be trained on a single A100 GPU
using 25k questions in 5 hours. This is a testament of the quality of our
Open-Platypus dataset, and opens opportunities for more improvements in the
field. Project page: https://platypus-llm.github.io
The Devil is in the Errors: Leveraging Large Language Models for
  Fine-grained Machine Translation Evaluation
Authors: name: Patrick Fernandes, name: Daniel Deutsch, name: Mara Finkelstein, name: Parker Riley, name: André F. T. Martins, name: Graham Neubig, name: Ankush Garg, name: Jonathan H. Clark, name: Markus Freitag, name: Orhan Firat
Published: 2023-08-14 17:17:21

Automatic evaluation of machine translation (MT) is a critical tool driving
the rapid iterative development of MT systems. While considerable progress has
been made on estimating a single scalar quality score, current metrics lack the
informativeness of more detailed schemes that annotate individual errors, such
as Multidimensional Quality Metrics (MQM). In this paper, we help fill this gap
by proposing AutoMQM, a prompting technique which leverages the reasoning and
in-context learning capabilities of large language models (LLMs) and asks them
to identify and categorize errors in translations. We start by evaluating
recent LLMs, such as PaLM and PaLM-2, through simple score prediction
prompting, and we study the impact of labeled data through in-context learning
and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that
it improves performance compared to just prompting for scores (with
particularly large gains for larger models) while providing interpretability
through error spans that align with human annotations.
OctoPack: Instruction Tuning Code Large Language Models
Authors: name: Niklas Muennighoff, name: Qian Liu, name: Armel Zebaze, name: Qinkai Zheng, name: Binyuan Hui, name: Terry Yue Zhuo, name: Swayam Singh, name: Xiangru Tang, name: Leandro von Werra, name: Shayne Longpre
Published: 2023-08-14 13:53:54

Finetuning large language models (LLMs) on instructions leads to vast
performance improvements on natural language tasks. We apply instruction tuning
using code, leveraging the natural structure of Git commits, which pair code
changes with human instructions. We compile CommitPack: 4 terabytes of Git
commits across 350 programming languages. We benchmark CommitPack against other
natural and synthetic code instructions (xP3x, Self-Instruct, OASST) on the 16B
parameter StarCoder model, and achieve state-of-the-art performance among
models not trained on OpenAI outputs, on the HumanEval Python benchmark (46.2%
pass@1). We further introduce HumanEvalPack, expanding the HumanEval benchmark
to a total of 3 coding tasks (Code Repair, Code Explanation, Code Synthesis)
across 6 languages (Python, JavaScript, Java, Go, C++, Rust). Our models,
OctoCoder and OctoGeeX, achieve the best performance across HumanEvalPack among
all permissive models, demonstrating CommitPack's benefits in generalizing to a
wider set of languages and natural coding tasks. Code, models and data are
freely available at https://github.com/bigcode-project/octopack.
CausalLM is not optimal for in-context learning
Authors: name: Nan Ding, name: Tomer Levinboim, name: Jialin Wu, name: Sebastian Goodman, name: Radu Soricut
Published: 2023-08-14 03:14:38

Recent empirical evidence indicates that transformer based in-context
learning performs better when using a prefix language model (prefixLM), in
which in-context samples can all attend to each other, compared to causal
language models (causalLM), which use auto-regressive attention that prohibits
in-context samples to attend to future samples. While this result is intuitive,
it is not understood from a theoretical perspective. In this paper we take a
theoretical approach and analyze the convergence behavior of prefixLM and
causalLM under a certain parameter construction. Our analysis shows that both
LM types converge to their stationary points at a linear rate, but that while
prefixLM converges to the optimal solution of linear regression, causalLM
convergence dynamics follows that of an online gradient descent algorithm,
which is not guaranteed to be optimal even as the number of samples grows
infinitely. We supplement our theoretical claims with empirical experiments
over synthetic and real tasks and using various types of transformers. Our
experiments verify that causalLM consistently underperforms prefixLM in all
settings.
VisIT-Bench: A Benchmark for Vision-Language Instruction Following
  Inspired by Real-World Use
Authors: name: Yonatan Bitton, name: Hritik Bansal, name: Jack Hessel, name: Rulin Shao, name: Wanrong Zhu, name: Anas Awadalla, name: Josh Gardner, name: Rohan Taori, name: Ludwig Schimdt
Published: 2023-08-12 15:27:51

We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for
evaluation of instruction-following vision-language models for real-world use.
Our starting point is curating 70 'instruction families' that we envision
instruction tuned vision-language models should be able to address. Extending
beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to
game playing and creative generation. Following curation, our dataset comprises
592 test queries, each with a human-authored instruction-conditioned caption.
These descriptions surface instruction-specific factors, e.g., for an
instruction asking about the accessibility of a storefront for wheelchair
users, the instruction-conditioned caption describes ramps/potential obstacles.
These descriptions enable 1) collecting human-verified reference outputs for
each instance; and 2) automatic evaluation of candidate multimodal generations
using a text-only LLM, aligning with human judgment. We quantify quality gaps
between models and references using both human and automatic evaluations; e.g.,
the top-performing instruction-following model wins against the GPT-4 reference
in just 27% of the comparison. VisIT-Bench is dynamic to participate,
practitioners simply submit their model's response on the project website;
Data, code and leaderboard is available at visit-bench.github.io.
Composable Function-preserving Expansions for Transformer Architectures
Authors: name: Andrea Gesmundo, name: Kaitlin Maile
Published: 2023-08-11 12:27:22

Training state-of-the-art neural networks requires a high cost in terms of
compute and time. Model scale is recognized to be a critical factor to achieve
and improve the state-of-the-art. Increasing the scale of a neural network
normally requires restarting from scratch by randomly initializing all the
parameters of the model, as this implies a change of architecture's parameters
that does not allow for a straightforward transfer of knowledge from smaller
size models. In this work, we propose six composable transformations to
incrementally increase the size of transformer-based neural networks while
preserving functionality, allowing to expand the capacity of the model as
needed. We provide proof of exact function preservation under minimal
initialization constraints for each transformation. The proposed methods may
enable efficient training pipelines for larger and more powerful models by
progressively expanding the architecture throughout training.
Improving Joint Speech-Text Representations Without Alignment
Authors: name: Cal Peyser, name: Zhong Meng, name: Ke Hu, name: Rohit Prabhavalkar, name: Andrew Rosenberg, name: Tara N. Sainath, name: Michael Picheny, name: Kyunghyun Cho
Published: 2023-08-11 13:28:48

The last year has seen astonishing progress in text-prompted image generation
premised on the idea of a cross-modal representation space in which the text
and image domains are represented jointly. In ASR, this idea has found
application as joint speech-text encoders that can scale to the capacities of
very large parameter models by being trained on both unpaired speech and text.
While these methods show promise, they have required special treatment of the
sequence-length mismatch inherent in speech and text, either by up-sampling
heuristics or an explicit alignment model. In this work, we offer evidence that
joint speech-text encoders naturally achieve consistent representations across
modalities by disregarding sequence length, and argue that consistency losses
could forgive length differences and simply assume the best alignment. We show
that such a loss improves downstream WER in both a large-parameter monolingual
and multilingual system.
Enhancing Network Management Using Code Generated by Large Language
  Models
Authors: name: Sathiya Kumaran Mani, name: Yajie Zhou, name: Kevin Hsieh, name: Santiago Segarra, name: Ranveer Chandra, name: Srikanth Kandula
Published: 2023-08-11 17:49:15

Analyzing network topologies and communication graphs plays a crucial role in
contemporary network management. However, the absence of a cohesive approach
leads to a challenging learning curve, heightened errors, and inefficiencies.
In this paper, we introduce a novel approach to facilitate a
natural-language-based network management experience, utilizing large language
models (LLMs) to generate task-specific code from natural language queries.
This method tackles the challenges of explainability, scalability, and privacy
by allowing network operators to inspect the generated code, eliminating the
need to share network data with LLMs, and concentrating on application-specific
requests combined with general program synthesis techniques. We design and
evaluate a prototype system using benchmark applications, showcasing high
accuracy, cost-effectiveness, and the potential for further enhancements using
complementary program synthesis techniques.
Self-Alignment with Instruction Backtranslation
Authors: name: Xian Li, name: Ping Yu, name: Chunting Zhou, name: Timo Schick, name: Luke Zettlemoyer, name: Omer Levy, name: Jason Weston, name: Mike Lewis
Published: 2023-08-11 17:47:54

We present a scalable method to build a high quality instruction following
language model by automatically labelling human-written text with corresponding
instructions. Our approach, named instruction backtranslation, starts with a
language model finetuned on a small amount of seed data, and a given web
corpus. The seed model is used to construct training examples by generating
instruction prompts for web documents (self-augmentation), and then selecting
high quality examples from among these candidates (self-curation). This data is
then used to finetune a stronger model. Finetuning LLaMa on two iterations of
our approach yields a model that outperforms all other LLaMa-based models on
the Alpaca leaderboard not relying on distillation data, demonstrating highly
effective self-alignment.
BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agents
Authors: name: Zhiwei Liu, name: Weiran Yao, name: Jianguo Zhang, name: Le Xue, name: Shelby Heinecke, name: Rithesh Murthy, name: Yihao Feng, name: Zeyuan Chen, name: Juan Carlos Niebles, name: Devansh Arpit, name: Ran Xu, name: Phil Mui, name: Huan Wang, name: Caiming Xiong, name: Silvio Savarese
Published: 2023-08-11 06:37:54

The massive successes of large language models (LLMs) encourage the emerging
exploration of LLM-augmented Autonomous Agents (LAAs). An LAA is able to
generate actions with its core LLM and interact with environments, which
facilitates the ability to resolve complex tasks by conditioning on past
interactions such as observations and actions. Since the investigation of LAA
is still very recent, limited explorations are available. Therefore, we provide
a comprehensive comparison of LAA in terms of both agent architectures and LLM
backbones. Additionally, we propose a new strategy to orchestrate multiple LAAs
such that each labor LAA focuses on one type of action, i.e. BOLAA,
where a controller manages the communication among multiple agents. We conduct
simulations on both decision-making and multi-step reasoning environments,
which comprehensively justify the capacity of LAAs. Our performance results
provide quantitative suggestions for designing LAA architectures and the
optimal choice of LLMs, as well as the compatibility of both. We release our
implementation code of LAAs to the public at
https://github.com/salesforce/BOLAA.
PIPPA: A Partially Synthetic Conversational Dataset
Authors: name: Tear Gosling, name: Alpin Dale, name: Yinhe Zheng
Published: 2023-08-11 00:33:26

With the emergence of increasingly powerful large language models, there is a
burgeoning interest in leveraging these models for casual conversation and
role-play applications. However, existing conversational and role-playing
datasets often fail to capture the diverse and nuanced interactions typically
exhibited by real-world role-play participants. To address this limitation and
contribute to the rapidly growing field, we introduce a partially-synthetic
dataset named PIPPA (Personal Interaction Pairs between People and AI). PIPPA
is a result of a community-driven crowdsourcing effort involving a group of
role-play enthusiasts. The dataset comprises over 1 million utterances that are
distributed across 26,000 conversation sessions and provides a rich resource
for researchers and AI developers to explore and refine conversational AI
systems in the context of role-play scenarios.
Follow Anything: Open-set detection, tracking, and following in
  real-time
Authors: name: Alaa Maalouf, name: Ninad Jadhav, name: Krishna Murthy Jatavallabhula, name: Makram Chahine, name: Daniel M. Vogt, name: Robert J. Wood, name: Antonio Torralba, name: Daniela Rus
Published: 2023-08-10 17:57:06

Tracking and following objects of interest is critical to several robotics
use cases, ranging from industrial automation to logistics and warehousing, to
healthcare and security. In this paper, we present a robotic system to detect,
track, and follow any object in real-time. Our approach, dubbed ``follow
anything'' (FAn), is an open-vocabulary and multimodal model -- it is not
restricted to concepts seen at training time and can be applied to novel
classes at inference time using text, images, or click queries. Leveraging rich
visual descriptors from large-scale pre-trained models (foundation models), FAn
can detect and segment objects by matching multimodal queries (text, images,
clicks) against an input image sequence. These detected and segmented objects
are tracked across image frames, all while accounting for occlusion and object
re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial
vehicle) and report its ability to seamlessly follow the objects of interest in
a real-time control loop. FAn can be deployed on a laptop with a lightweight
(6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To
enable rapid adoption, deployment, and extensibility, we open-source all our
code on our project webpage at https://github.com/alaamaalouf/FollowAnything .
We also encourage the reader the watch our 5-minutes explainer video in this
https://www.youtube.com/watch?v=6Mgt3EPytrw .