Orionnnn/hub-stats huggingface dataset:
Toward Grounded Social Reasoning
Authors: name: Minae Kwon, name: Hengyuan Hu, name: Vivek Myers, name: Siddharth Karamcheti, name: Anca Dragan, name: Dorsa Sadigh
Published: 2023-06-14 17:30:57

Consider a robot tasked with tidying a desk with a meticulously constructed
Lego sports car. A human may recognize that it is not socially appropriate to
disassemble the sports car and put it away as part of the "tidying". How can a
robot reach that conclusion? Although large language models (LLMs) have
recently been used to enable social reasoning, grounding this reasoning in the
real world has been challenging. To reason in the real world, robots must go
beyond passively querying LLMs and *actively gather information from the
environment* that is required to make the right decision. For instance, after
detecting that there is an occluded car, the robot may need to actively
perceive the car to know whether it is an advanced model car made out of Legos
or a toy car built by a toddler. We propose an approach that leverages an LLM
and vision language model (VLM) to help a robot actively perceive its
environment to perform grounded social reasoning. To evaluate our framework at
scale, we release the MessySurfaces dataset which contains images of 70
real-world surfaces that need to be cleaned. We additionally illustrate our
approach with a robot on 2 carefully designed surfaces. We find an average
12.9% improvement on the MessySurfaces benchmark and an average 15% improvement
on the robot experiments over baselines that do not use active perception. The
dataset, code, and videos of our approach can be found at
https://minaek.github.io/groundedsocialreasoning.
Language to Rewards for Robotic Skill Synthesis
Authors: name: Wenhao Yu, name: Nimrod Gileadi, name: Chuyuan Fu, name: Sean Kirmani, name: Kuang-Huei Lee, name: Montse Gonzalez Arenas, name: Hao-Tien Lewis Chiang, name: Tom Erez, name: Leonard Hasenclever, name: Jan Humplik, name: Brian Ichter, name: Ted Xiao, name: Peng Xu, name: Andy Zeng, name: Tingnan Zhang, name: Nicolas Heess, name: Dorsa Sadigh, name: Jie Tan, name: Yuval Tassa, name: Fei Xia
Published: 2023-06-14 17:27:10

Large language models (LLMs) have demonstrated exciting progress in acquiring
diverse new capabilities through in-context learning, ranging from logical
reasoning to code-writing. Robotics researchers have also explored using LLMs
to advance the capabilities of robotic control. However, since low-level robot
actions are hardware-dependent and underrepresented in LLM training corpora,
existing efforts in applying LLMs to robotics have largely treated LLMs as
semantic planners or relied on human-engineered control primitives to interface
with the robot. On the other hand, reward functions are shown to be flexible
representations that can be optimized for control policies to achieve diverse
tasks, while their semantic richness makes them suitable to be specified by
LLMs. In this work, we introduce a new paradigm that harnesses this realization
by utilizing LLMs to define reward parameters that can be optimized and
accomplish variety of robotic tasks. Using reward as the intermediate interface
generated by LLMs, we can effectively bridge the gap between high-level
language instructions or corrections to low-level robot actions. Meanwhile,
combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive
behavior creation experience where users can immediately observe the results
and provide feedback to the system. To systematically evaluate the performance
of our proposed method, we designed a total of 17 tasks for a simulated
quadruped robot and a dexterous manipulator robot. We demonstrate that our
proposed method reliably tackles 90% of the designed tasks, while a baseline
using primitive skills as the interface with Code-as-policies achieves 50% of
the tasks. We further validated our method on a real robot arm where complex
manipulation skills such as non-prehensile pushing emerge through our
interactive system.
Seeing the World through Your Eyes
Authors: name: Hadi Alzayer, name: Kevin Zhang, name: Brandon Feng, name: Christopher Metzler, name: Jia-Bin Huang
Published: 2023-06-15 17:59:59

The reflective nature of the human eye is an underappreciated source of
information about what the world around us looks like. By imaging the eyes of a
moving person, we can collect multiple views of a scene outside the camera's
direct line of sight through the reflections in the eyes. In this paper, we
reconstruct a 3D scene beyond the camera's line of sight using portrait images
containing eye reflections. This task is challenging due to 1) the difficulty
of accurately estimating eye poses and 2) the entangled appearance of the eye
iris and the scene reflections. Our method jointly refines the cornea poses,
the radiance field depicting the scene, and the observer's eye iris texture. We
further propose a simple regularization prior on the iris texture pattern to
improve reconstruction quality. Through various experiments on synthetic and
real-world captures featuring people with varied eye colors, we demonstrate the
feasibility of our approach to recover 3D scenes using eye reflections.
UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video
Authors: name: Zhi-Hao Lin, name: Bohan Liu, name: Yi-Ting Chen, name: David Forsyth, name: Jia-Bin Huang, name: Anand Bhattad, name: Shenlong Wang
Published: 2023-06-15 17:59:59

We show how to build a model that allows realistic, free-viewpoint renderings
of a scene under novel lighting conditions from video. Our method -- UrbanIR:
Urban Scene Inverse Rendering -- computes an inverse graphics representation
from the video. UrbanIR jointly infers shape, albedo, visibility, and sun and
sky illumination from a single video of unbounded outdoor scenes with unknown
lighting. UrbanIR uses videos from cameras mounted on cars (in contrast to many
views of the same points in typical NeRF-style estimation). As a result,
standard methods produce poor geometry estimates (for example, roofs), and
there are numerous ''floaters''. Errors in inverse graphics inference can
result in strong rendering artifacts. UrbanIR uses novel losses to control
these and other sources of error. UrbanIR uses a novel loss to make very good
estimates of shadow volumes in the original scene. The resulting
representations facilitate controllable editing, delivering photorealistic
free-viewpoint renderings of relit scenes and inserted objects. Qualitative
evaluation demonstrates strong improvements over the state-of-the-art.
AssistGPT: A General Multi-modal Assistant that can Plan, Execute,
  Inspect, and Learn
Authors: name: Difei Gao, name: Lei Ji, name: Luowei Zhou, name: Kevin Qinghong Lin, name: Joya Chen, name: Zihan Fan, name: Mike Zheng Shou
Published: 2023-06-14 17:12:56

Recent research on Large Language Models (LLMs) has led to remarkable
advancements in general NLP AI assistants. Some studies have further explored
the use of LLMs for planning and invoking models or APIs to address more
general multi-modal user queries. Despite this progress, complex visual-based
tasks still remain challenging due to the diverse nature of visual tasks. This
diversity is reflected in two aspects: 1) Reasoning paths. For many real-life
applications, it is hard to accurately decompose a query simply by examining
the query itself. Planning based on the specific visual content and the results
of each step is usually required. 2) Flexible inputs and intermediate results.
Input forms could be flexible for in-the-wild cases, and involves not only a
single image or video but a mixture of videos and images, e.g., a user-view
image with some reference videos. Besides, a complex reasoning process will
also generate diverse multimodal intermediate results, e.g., video narrations,
segmented video clips, etc. To address such general cases, we propose a
multi-modal AI assistant, AssistGPT, with an interleaved code and language
reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate
LLMs with various tools. Specifically, the Planner is capable of using natural
language to plan which tool in Executor should do next based on the current
reasoning progress. Inspector is an efficient memory manager to assist the
Planner to feed proper visual information into a specific tool. Finally, since
the entire reasoning process is complex and flexible, a Learner is designed to
enable the model to autonomously explore and discover the optimal solution. We
conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving
state-of-the-art results. Moreover, showcases demonstrate the ability of our
system to handle questions far more complex than those found in the benchmarks.
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Authors: name: Ziyang Luo, name: Can Xu, name: Pu Zhao, name: Qingfeng Sun, name: Xiubo Geng, name: Wenxiang Hu, name: Chongyang Tao, name: Jing Ma, name: Qingwei Lin, name: Daxin Jiang
Published: 2023-06-14 15:18:48

Code Large Language Models (Code LLMs), such as StarCoder, have demonstrated
exceptional performance in code-related tasks. However, most existing models
are solely pre-trained on extensive raw code data without instruction
fine-tuning. In this paper, we introduce WizardCoder, which empowers Code LLMs
with complex instruction fine-tuning, by adapting the Evol-Instruct method to
the domain of code. Through comprehensive experiments on four prominent code
generation benchmarks, namely HumanEval, HumanEval+, MBPP, and DS-1000, we
unveil the exceptional capabilities of our model. It surpasses all other
open-source Code LLMs by a substantial margin. Moreover, our model even
outperforms the largest closed LLMs, Anthropic's Claude and Google's Bard, on
HumanEval and HumanEval+. Our code, model weights, and data are public at
https://github.com/nlpxucan/WizardLM
AVIS: Autonomous Visual Information Seeking with Large Language Models
Authors: name: Ziniu Hu, name: Ahmet Iscen, name: Chen Sun, name: Kai-Wei Chang, name: Yizhou Sun, name: David A Ross, name: Cordelia Schmid, name: Alireza Fathi
Published: 2023-06-13 20:50:22

In this paper, we propose an autonomous information seeking visual question
answering framework, AVIS. Our method leverages a Large Language Model (LLM) to
dynamically strategize the utilization of external tools and to investigate
their outputs, thereby acquiring the indispensable knowledge needed to provide
answers to the posed questions. Responding to visual questions that necessitate
external knowledge, such as "What event is commemorated by the building
depicted in this image?", is a complex task. This task presents a combinatorial
search space that demands a sequence of actions, including invoking APIs,
analyzing their responses, and making informed decisions. We conduct a user
study to collect a variety of instances of human decision-making when faced
with this task. This data is then used to design a system comprised of three
components: an LLM-powered planner that dynamically determines which tool to
use next, an LLM-powered reasoner that analyzes and extracts key information
from the tool outputs, and a working memory component that retains the acquired
information throughout the process. The collected user behavior serves as a
guide for our system in two key ways. First, we create a transition graph by
analyzing the sequence of decisions made by users. This graph delineates
distinct states and confines the set of actions available at each state.
Second, we use examples of user decision-making to provide our LLM-powered
planner and reasoner with relevant contextual instances, enhancing their
capacity to make informed decisions. We show that AVIS achieves
state-of-the-art results on knowledge-intensive visual question answering
benchmarks such as Infoseek and OK-VQA.
Tune As You Scale: Hyperparameter Optimization For Compute Efficient
  Training
Authors: name: Abraham J. Fetterman, name: Ellie Kitanidis, name: Joshua Albrecht, name: Zachary Polizzi, name: Bryden Fogelman, name: Maksis Knutins, name: Bartosz Wróblewski, name: James B. Simon, name: Kanjun Qiu
Published: 2023-06-13 18:22:24

Hyperparameter tuning of deep learning models can lead to order-of-magnitude
performance gains for the same amount of compute. Despite this, systematic
tuning is uncommon, particularly for large models, which are expensive to
evaluate and tend to have many hyperparameters, necessitating difficult
judgment calls about tradeoffs, budgets, and search bounds. To address these
issues and propose a practical method for robustly tuning large models, we
present Cost-Aware Pareto Region Bayesian Search (CARBS), a Bayesian
optimization algorithm that performs local search around the performance-cost
Pareto frontier. CARBS does well even in unbounded search spaces with many
hyperparameters, learns scaling relationships so that it can tune models even
as they are scaled up, and automates much of the "black magic" of tuning. Among
our results, we effectively solve the entire ProcGen benchmark just by tuning a
simple baseline (PPO, as provided in the original ProcGen paper). We also
reproduce the model size vs. training tokens scaling result from the Chinchilla
project (Hoffmann et al. 2022), while simultaneously discovering scaling laws
for every other hyperparameter, via an easy automated process that uses
significantly less compute and is applicable to any deep learning problem (not
just language models).
Knowledge Distillation of Large Language Models
Authors: name: Yuxian Gu, name: Li Dong, name: Furu Wei, name: Minlie Huang
Published: 2023-06-14 14:44:03

Knowledge Distillation (KD) is a promising technique for reducing the high
computational demand of large language models (LLMs). However, previous KD
methods are primarily applied to white-box classification models or training
small models to imitate black-box model APIs like ChatGPT. How to effectively
distill the knowledge from white-box generative LLMs is still under-explored,
which becomes more and more important with the prosperity of LLMs. In this
work, we propose MiniLLM that distills smaller language models from generative
larger language models. We first replace the forward Kullback-Leibler
divergence (KLD) objective in the standard KD approaches with reverse KLD,
which is more suitable for KD on generative language models, to prevent the
student model from overestimating the low-probability regions of the teacher
distribution. Then, we derive an effective optimization approach to learn this
objective. Extensive experiments in the instruction-following setting show that
the MiniLLM models generate more precise responses with the higher overall
quality, lower exposure bias, better calibration, and higher long-text
generation performance. Our method is also scalable for different model
families with 120M to 13B parameters. We will release our code and model
checkpoints at https://aka.ms/MiniLLM.
Exploring the MIT Mathematics and EECS Curriculum Using Large Language
  Models
Authors: name: Sarah J. Zhang, name: Samuel Florin, name: Ariel N. Lee, name: Eamon Niknafs, name: Andrei Marginean, name: Annie Wang, name: Keith Tyser, name: Zad Chin, name: Yann Hicke, name: Nikhil Singh, name: Madeleine Udell, name: Yoon Kim, name: Tonio Buonassisi, name: Armando Solar-Lezama, name: Iddo Drori
Published: 2023-06-15 09:48:14

We curate a comprehensive dataset of 4,550 questions and solutions from
problem sets, midterm exams, and final exams across all MIT Mathematics and
Electrical Engineering and Computer Science (EECS) courses required for
obtaining a degree. We evaluate the ability of large language models to fulfill
the graduation requirements for any MIT major in Mathematics and EECS. Our
results demonstrate that GPT-3.5 successfully solves a third of the entire MIT
curriculum, while GPT-4, with prompt engineering, achieves a perfect solve rate
on a test set excluding questions based on images. We fine-tune an open-source
large language model on this dataset. We employ GPT-4 to automatically grade
model responses, providing a detailed performance breakdown by course,
question, and answer type. By embedding questions in a low-dimensional space,
we explore the relationships between questions, topics, and classes and
discover which questions and classes are required for solving other questions
and classes through few-shot learning. Our analysis offers valuable insights
into course prerequisites and curriculum design, highlighting language models'
potential for learning and improving Mathematics and EECS education.
h2oGPT: Democratizing Large Language Models
Authors: name: Arno Candel, name: Jon McKinney, name: Philipp Singer, name: Pascal Pfeiffer, name: Maximilian Jeblick, name: Prithvi Prabhu, name: Jeff Gambera, name: Mark Landry, name: Shivam Bansal, name: Ryan Chesler, name: Chun Ming Lee, name: Marcos V. Conde, name: Pasha Stetsenko, name: Olivier Grellier, name: SriSatish Ambati
Published: 2023-06-13 22:19:53

Foundation Large Language Models (LLMs) such as GPT-4 represent a revolution
in AI due to their real-world applications though natural language processing.
However, they also pose many significant risks such as the presence of biased,
private, or harmful text, and the unauthorized inclusion of copyrighted
material.
  We introduce h2oGPT, a suite of open-source code repositories for the
creation and use of Large Language Models (LLMs) based on Generative Pretrained
Transformers (GPTs). The goal of this project is to create the world's best
truly open-source alternative to closed-source GPTs. In collaboration with and
as part of the incredible and unstoppable open-source community, we open-source
several fine-tuned h2oGPT models from 7 to 40 Billion parameters, ready for
commercial use under fully permissive Apache 2.0 licenses. Included in our
release is 100% private document search using natural language.
  Open-source language models help boost AI development and make it more
accessible and trustworthy. They lower entry hurdles, allowing people and
groups to tailor these models to their needs. This openness increases
innovation, transparency, and fairness. An open-source strategy is needed to
share AI benefits fairly, and H2O.ai will continue to democratize AI and LLMs.
Galactic: Scaling End-to-End Reinforcement Learning for Rearrangement at
  100k Steps-Per-Second
Authors: name: Vincent-Pierre Berges, name: Andrew Szot, name: Devendra Singh Chaplot, name: Aaron Gokaslan, name: Roozbeh Mottaghi, name: Dhruv Batra, name: Eric Undersander
Published: 2023-06-13 05:53:23

We present Galactic, a large-scale simulation and reinforcement-learning (RL)
framework for robotic mobile manipulation in indoor environments. Specifically,
a Fetch robot (equipped with a mobile base, 7DoF arm, RGBD camera, egomotion,
and onboard sensing) is spawned in a home environment and asked to rearrange
objects - by navigating to an object, picking it up, navigating to a target
location, and then placing the object at the target location.
  Galactic is fast. In terms of simulation speed (rendering + physics),
Galactic achieves over 421,000 steps-per-second (SPS) on an 8-GPU node, which
is 54x faster than Habitat 2.0 (7699 SPS). More importantly, Galactic was
designed to optimize the entire rendering + physics + RL interplay since any
bottleneck in the interplay slows down training. In terms of simulation+RL
speed (rendering + physics + inference + learning), Galactic achieves over
108,000 SPS, which 88x faster than Habitat 2.0 (1243 SPS).
  These massive speed-ups not only drastically cut the wall-clock training time
of existing experiments, but also unlock an unprecedented scale of new
experiments. First, Galactic can train a mobile pick skill to >80% accuracy in
under 16 minutes, a 100x speedup compared to the over 24 hours it takes to
train the same skill in Habitat 2.0. Second, we use Galactic to perform the
largest-scale experiment to date for rearrangement using 5B steps of experience
in 46 hours, which is equivalent to 20 years of robot experience. This scaling
results in a single neural network composed of task-agnostic components
achieving 85% success in GeometricGoal rearrangement, compared to 0% success
reported in Habitat 2.0 for the same approach. The code is available at
github.com/facebookresearch/galactic.
ATT3D: Amortized Text-to-3D Object Synthesis
Authors: name: Jonathan Lorraine, name: Kevin Xie, name: Xiaohui Zeng, name: Chen-Hsuan Lin, name: Towaki Takikawa, name: Nicholas Sharp, name: Tsung-Yi Lin, name: Ming-Yu Liu, name: Sanja Fidler, name: James Lucas
Published: 2023-06-06 17:59:10

Text-to-3D modelling has seen exciting progress by combining generative
text-to-image models with image-to-3D methods like Neural Radiance Fields.
DreamFusion recently achieved high-quality results but requires a lengthy,
per-prompt optimization to create 3D objects. To address this, we amortize
optimization over text prompts by training on many prompts simultaneously with
a unified model, instead of separately. With this, we share computation across
a prompt set, training in less time than per-prompt optimization. Our framework
- Amortized text-to-3D (ATT3D) - enables knowledge-sharing between prompts to
generalize to unseen setups and smooth interpolations between text for novel
assets and simple animations.
Neural Scene Chronology
Authors: name: Haotong Lin, name: Qianqian Wang, name: Ruojin Cai, name: Sida Peng, name: Hadar Averbuch-Elor, name: Xiaowei Zhou, name: Noah Snavely
Published: 2023-06-13 17:59:58

In this work, we aim to reconstruct a time-varying 3D model, capable of
rendering photo-realistic renderings with independent control of viewpoint,
illumination, and time, from Internet photos of large-scale landmarks. The core
challenges are twofold. First, different types of temporal changes, such as
illumination and changes to the underlying scene itself (such as replacing one
graffiti artwork with another) are entangled together in the imagery. Second,
scene-level temporal changes are often discrete and sporadic over time, rather
than continuous. To tackle these problems, we propose a new scene
representation equipped with a novel temporal step function encoding method
that can model discrete scene-level content changes as piece-wise constant
functions over time. Specifically, we represent the scene as a space-time
radiance field with a per-image illumination embedding, where
temporally-varying scene changes are encoded using a set of learned step
functions. To facilitate our task of chronology reconstruction from Internet
imagery, we also collect a new dataset of four scenes that exhibit various
changes over time. We demonstrate that our method exhibits state-of-the-art
view synthesis results on this dataset, while achieving independent control of
viewpoint, time, and illumination.