Orionnnn/hub-stats huggingface dataset:
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural
  Feature Fields
Authors: name: Yanjie Ze, name: Ge Yan, name: Yueh-Hua Wu, name: Annabella Macaluso, name: Yuying Ge, name: Jianglong Ye, name: Nicklas Hansen, name: Li Erran Li, name: Xiaolong Wang
Published: 2023-08-31 17:52:10

It is a long-standing problem in robotics to develop agents capable of
executing diverse manipulation tasks from visual observations in unstructured
real-world environments. To achieve this goal, the robot needs to have a
comprehensive understanding of the 3D structure and semantics of the scene. In
this work, we present GNFactor, a visual behavior cloning agent for
multi-task robotic manipulation with Generalizable Neural
feature Fields. GNFactor jointly optimizes a generalizable neural
field (GNF) as a reconstruction module and a Perceiver Transformer as a
decision-making module, leveraging a shared deep 3D voxel representation. To
incorporate semantics in 3D, the reconstruction module utilizes a
vision-language foundation model (e.g., Stable Diffusion) to distill
rich semantic information into the deep 3D voxel. We evaluate GNFactor on 3
real robot tasks and perform detailed ablations on 10 RLBench tasks with a
limited number of demonstrations. We observe a substantial improvement of
GNFactor over current state-of-the-art methods in seen and unseen tasks,
demonstrating the strong generalization ability of GNFactor. Our project
website is https://yanjieze.com/GNFactor/ .
Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size
  HD Images
Authors: name: Qingping Zheng, name: Yuanfan Guo, name: Jiankang Deng, name: Jianhua Han, name: Ying Li, name: Songcen Xu, name: Hang Xu
Published: 2023-08-31 09:27:56

Stable diffusion, a generative model used in text-to-image synthesis,
frequently encounters resolution-induced composition problems when generating
images of varying sizes. This issue primarily stems from the model being
trained on pairs of single-scale images and their corresponding text
descriptions. Moreover, direct training on images of unlimited sizes is
unfeasible, as it would require an immense number of text-image pairs and
entail substantial computational expenses. To overcome these challenges, we
propose a two-stage pipeline named Any-Size-Diffusion (ASD), designed to
efficiently generate well-composed images of any size, while minimizing the
need for high-memory GPU resources. Specifically, the initial stage, dubbed Any
Ratio Adaptability Diffusion (ARAD), leverages a selected set of images with a
restricted range of ratios to optimize the text-conditional diffusion model,
thereby improving its ability to adjust composition to accommodate diverse
image sizes. To support the creation of images at any desired size, we further
introduce a technique called Fast Seamless Tiled Diffusion (FSTD) at the
subsequent stage. This method allows for the rapid enlargement of the ASD
output to any high-resolution size, avoiding seaming artifacts or memory
overloads. Experimental results on the LAION-COCO and MM-CelebA-HQ benchmarks
demonstrate that ASD can produce well-structured images of arbitrary sizes,
cutting down the inference time by 2x compared to the traditional tiled
algorithm.
MVDream: Multi-view Diffusion for 3D Generation
Authors: name: Yichun Shi, name: Peng Wang, name: Jianglong Ye, name: Mai Long, name: Kejie Li, name: Xiao Yang
Published: 2023-08-31 07:49:06

We propose MVDream, a multi-view diffusion model that is able to generate
geometrically consistent multi-view images from a given text prompt. By
leveraging image diffusion models pre-trained on large-scale web datasets and a
multi-view dataset rendered from 3D assets, the resulting multi-view diffusion
model can achieve both the generalizability of 2D diffusion and the consistency
of 3D data. Such a model can thus be applied as a multi-view prior for 3D
generation via Score Distillation Sampling, where it greatly improves the
stability of existing 2D-lifting methods by solving the 3D consistency problem.
Finally, we show that the multi-view diffusion model can also be fine-tuned
under a few shot setting for personalized 3D generation, i.e. DreamBooth3D
application, where the consistency can be maintained after learning the subject
identity.
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122
  Language Variants
Authors: name: Lucas Bandarkar, name: Davis Liang, name: Benjamin Muller, name: Mikel Artetxe, name: Satya Narayan Shukla, name: Donald Husa, name: Naman Goyal, name: Abhinandan Krishnan, name: Luke Zettlemoyer, name: Madian Khabsa
Published: 2023-08-31 17:43:08

We present Belebele, a multiple-choice machine reading comprehension (MRC)
dataset spanning 122 language variants. Significantly expanding the language
coverage of natural language understanding (NLU) benchmarks, this dataset
enables the evaluation of text models in high-, medium-, and low-resource
languages. Each question is based on a short passage from the Flores-200
dataset and has four multiple-choice answers. The questions were carefully
curated to discriminate between models with different levels of general
language comprehension. The English dataset on its own proves difficult enough
to challenge state-of-the-art language models. Being fully parallel, this
dataset enables direct comparison of model performance across all languages. We
use this dataset to evaluate the capabilities of multilingual masked language
models (MLMs) and large language models (LLMs). We present extensive results
and find that despite significant cross-lingual transfer in English-centric
LLMs, much smaller MLMs pretrained on balanced multilingual data still
understand far more languages. We also observe that larger vocabulary size and
conscious vocabulary construction correlate with better performance on
low-resource languages. Overall, Belebele opens up new avenues for evaluating
and analyzing the multilingual capabilities of NLP systems.
Can Programming Languages Boost Each Other via Instruction Tuning?
Authors: name: Daoguang Zan, name: Ailun Yu, name: Bo Shen, name: Jiaxin Zhang, name: Taihong Chen, name: Bing Geng, name: Bei Chen, name: Jichuan Ji, name: Yafen Yao, name: Yongji Wang, name: Qianxiang Wang
Published: 2023-08-31 15:53:51

When human programmers have mastered a programming language, it would be
easier when they learn a new programming language. In this report, we focus on
exploring whether programming languages can boost each other during the
instruction fine-tuning phase of code large language models. We conduct
extensive experiments of 8 popular programming languages (Python, JavaScript,
TypeScript, C, C++, Java, Go, HTML) on StarCoder. Results demonstrate that
programming languages can significantly improve each other. For example,
CodeM-Python 15B trained on Python is able to increase Java by an absolute
17.95% pass@1 on HumanEval-X. More surprisingly, we found that CodeM-HTML 7B
trained on the HTML corpus can improve Java by an absolute 15.24% pass@1. Our
training data is released at https://github.com/NL2Code/CodeM.
BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
  Pragmatic Knowledge
Authors: name: Xiangru Tang, name: Bill Qian, name: Rick Gao, name: Jiakang Chen, name: Xinyun Chen, name: Mark Gerstein
Published: 2023-08-31 04:52:58

Pre-trained language models like ChatGPT have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks. Moreover, in bioinformatics, generating
functional programs poses additional notable challenges due to the amount of
domain knowledge, the need for complicated data operations, and intricate
functional dependencies between the operations. Here, we present BioCoder, a
benchmark developed to evaluate existing pre-trained models in generating
bioinformatics code. In relation to function-code generation, BioCoder covers
potential package dependencies, class declarations, and global variables. It
incorporates 1026 functions and 1243 methods in Python and Java from GitHub and
253 examples from the Rosalind Project. BioCoder incorporates a fuzz-testing
framework for evaluation, and we have applied it to evaluate many models
including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+,
InstructCodeT5+, and ChatGPT. Our detailed analysis of these models emphasizes
the importance of domain knowledge, pragmatic code generation, and contextual
understanding. Our dataset, benchmark, Docker images, and scripts required for
testing are all available at https://github.com/gersteinlab/biocoder.
WeatherBench 2: A benchmark for the next generation of data-driven
  global weather models
Authors: name: Stephan Rasp, name: Stephan Hoyer, name: Alexander Merose, name: Ian Langmore, name: Peter Battaglia, name: Tyler Russel, name: Alvaro Sanchez-Gonzalez, name: Vivian Yang, name: Rob Carver, name: Shreya Agrawal, name: Matthew Chantry, name: Zied Ben Bouallegue, name: Peter Dueben, name: Carla Bromberg, name: Jared Sisk, name: Luke Barrington, name: Aaron Bell, name: Fei Sha
Published: 2023-08-29 18:32:08

WeatherBench 2 is an update to the global, medium-range (1-14 day) weather
forecasting benchmark proposed by Rasp et al. (2020), designed with the aim to
accelerate progress in data-driven weather modeling. WeatherBench 2 consists of
an open-source evaluation framework, publicly available training, ground truth
and baseline data as well as a continuously updated website with the latest
metrics and state-of-the-art models:
https://sites.research.google/weatherbench. This paper describes the design
principles of the evaluation framework and presents results for current
state-of-the-art physical and data-driven weather models. The metrics are based
on established practices for evaluating weather forecasts at leading
operational weather centers. We define a set of headline scores to provide an
overview of model performance. In addition, we also discuss caveats in the
current evaluation setup and challenges for the future of data-driven weather
forecasting.
Learning Vision-based Pursuit-Evasion Robot Policies
Authors: name: Andrea Bajcsy, name: Antonio Loquercio, name: Ashish Kumar, name: Jitendra Malik
Published: 2023-08-30 17:59:05

Learning strategic robot behavior -- like that required in pursuit-evasion
interactions -- under real-world constraints is extremely challenging. It
requires exploiting the dynamics of the interaction, and planning through both
physical state and latent intent uncertainty. In this paper, we transform this
intractable problem into a supervised learning problem, where a
fully-observable robot policy generates supervision for a partially-observable
one. We find that the quality of the supervision signal for the
partially-observable pursuer policy depends on two key factors: the balance of
diversity and optimality of the evader's behavior and the strength of the
modeling assumptions in the fully-observable policy. We deploy our policy on a
physical quadruped robot with an RGB-D camera on pursuit-evasion interactions
in the wild. Despite all the challenges, the sensing constraints bring about
creativity: the robot is pushed to gather information when uncertain, predict
intent from noisy measurements, and anticipate in order to intercept. Project
webpage: https://abajcsy.github.io/vision-based-pursuit/
RoboTAP: Tracking Arbitrary Points for Few-Shot Visual Imitation
Authors: name: Mel Vecerik, name: Carl Doersch, name: Yi Yang, name: Todor Davchev, name: Yusuf Aytar, name: Guangyao Zhou, name: Raia Hadsell, name: Lourdes Agapito, name: Jon Scholz
Published: 2023-08-30 11:57:04

For robots to be useful outside labs and specialized factories we need a way
to teach them new useful behaviors quickly. Current approaches lack either the
generality to onboard new tasks without task-specific engineering, or else lack
the data-efficiency to do so in an amount of time that enables practical use.
In this work we explore dense tracking as a representational vehicle to allow
faster and more general learning from demonstration. Our approach utilizes
Track-Any-Point (TAP) models to isolate the relevant motion in a demonstration,
and parameterize a low-level controller to reproduce this motion across changes
in the scene configuration. We show this results in robust robot policies that
can solve complex object-arrangement tasks such as shape-matching, stacking,
and even full path-following tasks such as applying glue and sticking objects
together, all from demonstrations that can be collected in minutes.
Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open
  Generative Large Language Models
Authors: name: Neha Sengupta, name: Sunil Kumar Sahu, name: Bokang Jia, name: Satheesh Katipomu, name: Haonan Li, name: Fajri Koto, name: Osama Mohammed Afzal, name: Samta Kamboj, name: Onkar Pandit, name: Rahul Pal, name: Lalit Pradhan, name: Zain Muhammad Mujahid, name: Massa Baali, name: Alham Fikri Aji, name: Zhengzhong Liu, name: Andy Hock, name: Andrew Feldman, name: Jonathan Lee, name: Andrew Jackson, name: Preslav Nakov, name: Timothy Baldwin, name: Eric Xing
Published: 2023-08-30 17:07:17

We introduce Jais and Jais-chat, new state-of-the-art Arabic-centric
foundation and instruction-tuned open generative large language models (LLMs).
The models are based on the GPT-3 decoder-only architecture and are pretrained
on a mixture of Arabic and English texts, including source code in various
programming languages. With 13 billion parameters, they demonstrate better
knowledge and reasoning capabilities in Arabic than any existing open Arabic
and multilingual models by a sizable margin, based on extensive evaluation.
Moreover, the models are competitive in English compared to English-centric
open models of similar size, despite being trained on much less English data.
We provide a detailed description of the training, the tuning, the safety
alignment, and the evaluation of the models. We release two open versions of
the model -- the foundation Jais model, and an instruction-tuned Jais-chat
variant -- with the aim of promoting research on Arabic LLMs. Available at
https://huggingface.co/inception-mbzuai/jais-13b-chat
LM-Infinite: Simple On-the-Fly Length Generalization for Large Language
  Models
Authors: name: Chi Han, name: Qifan Wang, name: Wenhan Xiong, name: Yu Chen, name: Heng Ji, name: Sinong Wang
Published: 2023-08-30 16:47:51

In recent years, there have been remarkable advancements in the performance
of Transformer-based Large Language Models (LLMs) across various domains. As
these LLMs are deployed for increasingly complex tasks, they often face the
needs to conduct longer reasoning processes or understanding larger contexts.
In these situations, the length generalization failure of LLMs on long
sequences become more prominent. Most pre-training schemes truncate training
sequences to a fixed length (such as 2048 for LLaMa). LLMs often struggle to
generate fluent texts, let alone carry out downstream tasks, after longer
contexts, even with relative positional encoding which is designed to cope with
this problem. Common solutions such as finetuning on longer corpora often
involves daunting hardware and time costs and requires careful training process
design. To more efficiently leverage the generation capacity of existing LLMs,
we theoretically and empirically investigate the main out-of-distribution (OOD)
factors contributing to this problem. Inspired by this diagnosis, we propose a
simple yet effective solution for on-the-fly length generalization,
LM-Infinite, which involves only a Lambda-shaped attention mask and a
distance limit while requiring no parameter updates or learning. We find it
applicable to a variety of LLMs using relative-position encoding methods.
LM-Infinite is computational efficient with O(n) time and space, and
demonstrates consistent fluency and generation quality to as long as 32k tokens
on ArXiv and OpenWebText2 datasets, with 2.72x decoding speedup. On downstream
task such as passkey retrieval, it continues to work on inputs much longer than
training lengths where vanilla models fail immediately.
LLaSM: Large Language and Speech Model
Authors: name: Yu Shu, name: Siwei Dong, name: Guangyao Chen, name: Wenhao Huang, name: Ruihua Zhang, name: Daochen Shi, name: Qiqi Xiang, name: Yemin Shi
Published: 2023-08-30 10:12:39

Multi-modal large language models have garnered significant interest
recently. Though, most of the works focus on vision-language multi-modal models
providing strong capabilities in following vision-and-language instructions.
However, we claim that speech is also an important modality through which
humans interact with the world. Hence, it is crucial for a general-purpose
assistant to be able to follow multi-modal speech-and-language instructions. In
this work, we propose Large Language and Speech Model (LLaSM). LLaSM is an
end-to-end trained large multi-modal speech-language model with cross-modal
conversational abilities, capable of following speech-and-language
instructions. Our early experiments show that LLaSM demonstrates a more
convenient and natural way for humans to interact with artificial intelligence.
Specifically, we also release a large Speech Instruction Following dataset
LLaSM-Audio-Instructions. Code and demo are available at
https://github.com/LinkSoul-AI/LLaSM and
https://huggingface.co/spaces/LinkSoul/LLaSM. The LLaSM-Audio-Instructions
dataset is available at
https://huggingface.co/datasets/LinkSoul/LLaSM-Audio-Instructions.
ORES: Open-vocabulary Responsible Visual Synthesis
Authors: name: Minheng Ni, name: Chenfei Wu, name: Xiaodong Wang, name: Shengming Yin, name: Lijuan Wang, name: Zicheng Liu, name: Nan Duan
Published: 2023-08-26 06:47:34

Avoiding synthesizing specific visual concepts is an essential challenge in
responsible visual synthesis. However, the visual concept that needs to be
avoided for responsible visual synthesis tends to be diverse, depending on the
region, context, and usage scenarios. In this work, we formalize a new task,
Open-vocabulary Responsible Visual Synthesis (ORES), where the synthesis model
is able to avoid forbidden visual concepts while allowing users to input any
desired content. To address this problem, we present a Two-stage Intervention
(TIN) framework. By introducing 1) rewriting with learnable instruction through
a large-scale language model (LLM) and 2) synthesizing with prompt intervention
on a diffusion synthesis model, it can effectively synthesize images avoiding
any concepts but following the user's query as much as possible. To evaluate on
ORES, we provide a publicly available dataset, baseline models, and benchmark.
Experimental results demonstrate the effectiveness of our method in reducing
risks of image generation. Our work highlights the potential of LLMs in
responsible visual synthesis. Our code and dataset is public available.
MedAlign: A Clinician-Generated Dataset for Instruction Following with
  Electronic Medical Records
Authors: name: Scott L. Fleming, name: Alejandro Lozano, name: William J. Haberkorn, name: Jenelle A. Jindal, name: Eduardo P. Reis, name: Rahul Thapa, name: Louis Blankemeier, name: Julian Z. Genkins, name: Ethan Steinberg, name: Ashwin Nayak, name: Birju S. Patel, name: Chia-Chun Chiang, name: Alison Callahan, name: Zepeng Huo, name: Sergios Gatidis, name: Scott J. Adams, name: Oluseyi Fayanju, name: Shreya J. Shah, name: Thomas Savage, name: Ethan Goh, name: Akshay S. Chaudhari, name: Nima Aghaeepour, name: Christopher Sharp, name: Michael A. Pfeffer, name: Percy Liang, name: Jonathan H. Chen, name: Keith E. Morse, name: Emma P. Brunskill, name: Jason A. Fries, name: Nigam H. Shah
Published: 2023-08-27 12:24:39

The ability of large language models (LLMs) to follow natural language
instructions with human-level fluency suggests many opportunities in healthcare
to reduce administrative burden and improve quality of care. However,
evaluating LLMs on realistic text generation tasks for healthcare remains
challenging. Existing question answering datasets for electronic health record
(EHR) data fail to capture the complexity of information needs and
documentation burdens experienced by clinicians. To address these challenges,
we introduce MedAlign, a benchmark dataset of 983 natural language instructions
for EHR data. MedAlign is curated by 15 clinicians (7 specialities), includes
clinician-written reference responses for 303 instructions, and provides 276
longitudinal EHRs for grounding instruction-response pairs. We used MedAlign to
evaluate 6 general domain LLMs, having clinicians rank the accuracy and quality
of each LLM response. We found high error rates, ranging from 35% (GPT-4) to
68% (MPT-7B-Instruct), and an 8.3% drop in accuracy moving from 32k to 2k
context lengths for GPT-4. Finally, we report correlations between clinician
rankings and automated natural language generation metrics as a way to rank
LLMs without human review. We make MedAlign available under a research data use
agreement to enable LLM evaluations on tasks aligned with clinician needs and
preferences.
Eventful Transformers: Leveraging Temporal Redundancy in Vision
  Transformers
Authors: name: Matthew Dutson, name: Yin Li, name: Mohit Gupta
Published: 2023-08-25 17:10:12

Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method for identifying
and re-processing only those tokens that have changed significantly over time.
Our proposed family of models, Eventful Transformers, can be converted from
existing Transformers (often without any re-training) and give adaptive control
over the compute cost at runtime. We evaluate our method on large-scale
datasets for video object detection (ImageNet VID) and action recognition
(EPIC-Kitchens 100). Our approach leads to significant computational savings
(on the order of 2-4x) with only minor reductions in accuracy.
Relighting Neural Radiance Fields with Shadow and Highlight Hints
Authors: name: Chong Zeng, name: Guojun Chen, name: Yue Dong, name: Pieter Peers, name: Hongzhi Wu, name: Xin Tong
Published: 2023-08-25 14:27:03

This paper presents a novel neural implicit radiance representation for free
viewpoint relighting from a small set of unstructured photographs of an object
lit by a moving point light source different from the view position. We express
the shape as a signed distance function modeled by a multi layer perceptron. In
contrast to prior relightable implicit neural representations, we do not
disentangle the different reflectance components, but model both the local and
global reflectance at each point by a second multi layer perceptron that, in
addition, to density features, the current position, the normal (from the
signed distace function), view direction, and light position, also takes shadow
and highlight hints to aid the network in modeling the corresponding high
frequency light transport effects. These hints are provided as a suggestion,
and we leave it up to the network to decide how to incorporate these in the
final relit result. We demonstrate and validate our neural implicit
representation on synthetic and real scenes exhibiting a wide variety of
shapes, material properties, and global illumination light transport.