Orionnnn/hub-stats huggingface dataset:
Introspective Tips: Large Language Model for In-Context Decision Making
Authors: name: Liting Chen, name: Lu Wang, name: Hang Dong, name: Yali Du, name: Jie Yan, name: Fangkai Yang, name: Shuang Li, name: Pu Zhao, name: Si Qin, name: Saravan Rajmohan, name: Qingwei Lin, name: Dongmei Zhang
Published: 2023-05-19 11:20:37

The emergence of large language models (LLMs) has substantially influenced
natural language processing, demonstrating exceptional results across various
tasks. In this study, we employ ``Introspective Tips" to facilitate LLMs in
self-optimizing their decision-making. By introspectively examining
trajectories, LLM refines its policy by generating succinct and valuable tips.
Our method enhances the agent's performance in both few-shot and zero-shot
learning situations by considering three essential scenarios: learning from the
agent's past experiences, integrating expert demonstrations, and generalizing
across diverse games. Importantly, we accomplish these improvements without
fine-tuning the LLM parameters; rather, we adjust the prompt to generalize
insights from the three aforementioned situations. Our framework not only
supports but also emphasizes the advantage of employing LLM in in-contxt
decision-making. Experiments involving over 100 games in TextWorld illustrate
the superior performance of our approach.
Empower Large Language Model to Perform Better on Industrial
  Domain-Specific Question Answering
Authors: name: Zezhong Wang, name: Fangkai Yang, name: Pu Zhao, name: Lu Wang, name: Jue Zhang, name: Mohit Garg, name: Qingwei Lin, name: Dongmei Zhang
Published: 2023-05-19 09:23:25

Large Language Model (LLM) has gained popularity and achieved remarkable
results in open-domain tasks, but its performance in real industrial
domain-specific scenarios is average since there is no specific knowledge in
it. This issue has attracted widespread attention, but there are few relevant
benchmarks available. In this paper, we provide a benchmark Question Answering
(QA) dataset named MSQA, which is about Microsoft products and IT technical
problems encountered by customers. This dataset contains industry
cloud-specific QA knowledge, which is not available for general LLM, so it is
well suited for evaluating methods aimed at improving domain-specific
capabilities of LLM. In addition, we propose a new model interaction paradigm
that can empower LLM to achieve better performance on domain-specific tasks
where it is not proficient. Extensive experiments demonstrate that the approach
following our model fusion framework outperforms the commonly used LLM with
retrieval methods.
Visualizing Linguistic Diversity of Text Datasets Synthesized by Large
  Language Models
Authors: name: Emily Reif, name: Minsuk Kahng, name: Savvas Petridis
Published: 2023-05-19 00:53:45

Large language models (LLMs) can be used to generate smaller, more refined
datasets via few-shot prompting for benchmarking, fine-tuning or other use
cases. However, understanding and evaluating these datasets is difficult, and
the failure modes of LLM-generated data are still not well understood.
Specifically, the data can be repetitive in surprising ways, not only
semantically but also syntactically and lexically. We present LinguisticLens, a
novel inter-active visualization tool for making sense of and analyzing
syntactic diversity of LLM-generated datasets. LinguisticLens clusters text
along syntactic, lexical, and semantic axes. It supports hierarchical
visualization of a text dataset, allowing users to quickly scan for an overview
and inspect individual examples. The live demo is available at
shorturl.at/zHOUV.
Counterfactuals for Design: A Model-Agnostic Method For Design
  Recommendations
Authors: name: Lyle Regenwetter, name: Yazan Abu Obaideh, name: Faez Ahmed
Published: 2023-05-18 21:10:58

We introduce Multi-Objective Counterfactuals for Design (MCD), a novel method
for counterfactual optimization in design problems. Counterfactuals are
hypothetical situations that can lead to a different decision or choice. In
this paper, the authors frame the counterfactual search problem as a design
recommendation tool that can help identify modifications to a design, leading
to better functional performance. MCD improves upon existing counterfactual
search methods by supporting multi-objective queries, which are crucial in
design problems, and by decoupling the counterfactual search and sampling
processes, thus enhancing efficiency and facilitating objective tradeoff
visualization. The paper demonstrates MCD's core functionality using a
two-dimensional test case, followed by three case studies of bicycle design
that showcase MCD's effectiveness in real-world design problems. In the first
case study, MCD excels at recommending modifications to query designs that can
significantly enhance functional performance, such as weight savings and
improvements to the structural safety factor. The second case study
demonstrates that MCD can work with a pre-trained language model to suggest
design changes based on a subjective text prompt effectively. Lastly, the
authors task MCD with increasing a query design's similarity to a target image
and text prompt while simultaneously reducing weight and improving structural
performance, demonstrating MCD's performance on a complex multimodal query.
Overall, MCD has the potential to provide valuable recommendations for
practitioners and design automation researchers looking for answers to their
``What if'' questions by exploring hypothetical design modifications and their
impact on multiple design objectives. The code, test problems, and datasets
used in the paper are available to the public at
decode.mit.edu/projects/counterfactuals/.
Comparing Machines and Children: Using Developmental Psychology
  Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
Authors: name: Eliza Kosoy, name: Emily Rose Reagan, name: Leslie Lai, name: Alison Gopnik, name: Danielle Krettek Cobb
Published: 2023-05-18 18:15:43

Developmental psychologists have spent decades devising experiments to test
the intelligence and knowledge of infants and children, tracing the origin of
crucial concepts and capacities. Moreover, experimental techniques in
developmental psychology have been carefully designed to discriminate the
cognitive capacities that underlie particular behaviors. We propose that using
classical experiments from child development is a particularly effective way to
probe the computational abilities of AI models, in general, and LLMs in
particular. First, the methodological techniques of developmental psychology,
such as the use of novel stimuli to control for past experience or control
conditions to determine whether children are using simple associations, can be
equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
in this way can tell us whether the information that is encoded in text is
sufficient to enable particular responses, or whether those responses depend on
other kinds of information, such as information from exploration of the
physical world. In this work we adapt classical developmental experiments to
evaluate the capabilities of LaMDA, a large language model from Google. We
propose a novel LLM Response Score (LRS) metric which can be used to evaluate
other language models, such as GPT. We find that LaMDA generates appropriate
responses that are similar to those of children in experiments involving social
understanding, perhaps providing evidence that knowledge of these domains is
discovered through language. On the other hand, LaMDA's responses in early
object and action understanding, theory of mind, and especially causal
reasoning tasks are very different from those of young children, perhaps
showing that these domains require more real-world, self-initiated exploration
and cannot simply be learned from patterns in language input.
Comparing Software Developers with ChatGPT: An Empirical Investigation
Authors: name: Nathalia Nascimento, name: Paulo Alencar, name: Donald Cowan
Published: 2023-05-19 17:25:54

The advent of automation in particular Software Engineering (SE) tasks has
transitioned from theory to reality. Numerous scholarly articles have
documented the successful application of Artificial Intelligence to address
issues in areas such as project management, modeling, testing, and development.
A recent innovation is the introduction of ChatGPT, an ML-infused chatbot,
touted as a resource proficient in generating programming codes and formulating
software testing strategies for developers and testers respectively. Although
there is speculation that AI-based computation can increase productivity and
even substitute software engineers in software development, there is currently
a lack of empirical evidence to verify this. Moreover, despite the primary
focus on enhancing the accuracy of AI systems, non-functional requirements
including energy efficiency, vulnerability, fairness (i.e., human bias), and
safety frequently receive insufficient attention. This paper posits that a
comprehensive comparison of software engineers and AI-based solutions,
considering various evaluation criteria, is pivotal in fostering human-machine
collaboration, enhancing the reliability of AI-based methods, and understanding
task suitability for humans or AI. Furthermore, it facilitates the effective
implementation of cooperative work structures and human-in-the-loop processes.
This paper conducts an empirical investigation, contrasting the performance of
software engineers and AI systems, like ChatGPT, across different evaluation
metrics. The empirical study includes a case of assessing ChatGPT-generated
code versus code produced by developers and uploaded in Leetcode.
LIMA: Less Is More for Alignment
Authors: name: Chunting Zhou, name: Pengfei Liu, name: Puxin Xu, name: Srini Iyer, name: Jiao Sun, name: Yuning Mao, name: Xuezhe Ma, name: Avia Efrat, name: Ping Yu, name: Lili Yu, name: Susan Zhang, name: Gargi Ghosh, name: Mike Lewis, name: Luke Zettlemoyer, name: Omer Levy
Published: 2023-05-18 17:45:22

Large language models are trained in two stages: (1) unsupervised pretraining
from raw text, to learn general-purpose representations, and (2) large scale
instruction tuning and reinforcement learning, to better align to end tasks and
user preferences. We measure the relative importance of these two stages by
training LIMA, a 65B parameter LLaMa language model fine-tuned with the
standard supervised loss on only 1,000 carefully curated prompts and responses,
without any reinforcement learning or human preference modeling. LIMA
demonstrates remarkably strong performance, learning to follow specific
response formats from only a handful of examples in the training data,
including complex queries that range from planning trip itineraries to
speculating about alternate history. Moreover, the model tends to generalize
well to unseen tasks that did not appear in the training data. In a controlled
human study, responses from LIMA are either equivalent or strictly preferred to
GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard
and 65% versus DaVinci003, which was trained with human feedback. Taken
together, these results strongly suggest that almost all knowledge in large
language models is learned during pretraining, and only limited instruction
tuning data is necessary to teach models to produce high quality output.
Chupa: Carving 3D Clothed Humans from Skinned Shape Priors using 2D
  Diffusion Probabilistic Models
Authors: name: Byungjun Kim, name: Patrick Kwon, name: Kwangho Lee, name: Myunggi Lee, name: Sookwan Han, name: Daesik Kim, name: Hanbyul Joo
Published: 2023-05-19 17:59:18

We propose a 3D generation pipeline that uses diffusion models to generate
realistic human digital avatars. Due to the wide variety of human identities,
poses, and stochastic details, the generation of 3D human meshes has been a
challenging problem. To address this, we decompose the problem into 2D normal
map generation and normal map-based 3D reconstruction. Specifically, we first
simultaneously generate realistic normal maps for the front and backside of a
clothed human, dubbed dual normal maps, using a pose-conditional diffusion
model. For 3D reconstruction, we ``carve'' the prior SMPL-X mesh to a detailed
3D mesh according to the normal maps through mesh optimization. To further
enhance the high-frequency details, we present a diffusion resampling scheme on
both body and facial regions, thus encouraging the generation of realistic
digital avatars. We also seamlessly incorporate a recent text-to-image
diffusion model to support text-based human identity control. Our method,
namely, Chupa, is capable of generating realistic 3D clothed humans with better
perceptual quality and identity variety.
Any-to-Any Generation via Composable Diffusion
Authors: name: Zineng Tang, name: Ziyi Yang, name: Chenguang Zhu, name: Michael Zeng, name: Mohit Bansal
Published: 2023-05-19 17:38:32

We present Composable Diffusion (CoDi), a novel generative model capable of
generating any combination of output modalities, such as language, image,
video, or audio, from any combination of input modalities. Unlike existing
generative AI systems, CoDi can generate multiple modalities in parallel and
its input is not limited to a subset of modalities like text or image. Despite
the absence of training datasets for many combinations of modalities, we
propose to align modalities in both the input and output space. This allows
CoDi to freely condition on any input combination and generate any group of
modalities, even if they are not present in the training data. CoDi employs a
novel composable generation strategy which involves building a shared
multimodal space by bridging alignment in the diffusion process, enabling the
synchronized generation of intertwined modalities, such as temporally aligned
video and audio. Highly customizable and flexible, CoDi achieves strong
joint-modality generation quality, and outperforms or is on par with the
unimodal state-of-the-art for single-modality synthesis. The project page with
demonstrations and code is at https://codi-gen.github.io
Cinematic Mindscapes: High-quality Video Reconstruction from Brain
  Activity
Authors: name: Zijiao Chen, name: Jiaxin Qing, name: Juan Helen Zhou
Published: 2023-05-19 13:44:25

Reconstructing human vision from brain activities has been an appealing task
that helps to understand our cognitive process. Even though recent research has
seen great success in reconstructing static images from non-invasive brain
recordings, work on recovering continuous visual experiences in the form of
videos is limited. In this work, we propose Mind-Video that learns
spatiotemporal information from continuous fMRI data of the cerebral cortex
progressively through masked brain modeling, multimodal contrastive learning
with spatiotemporal attention, and co-training with an augmented Stable
Diffusion model that incorporates network temporal inflation. We show that
high-quality videos of arbitrary frame rates can be reconstructed with
Mind-Video using adversarial guidance. The recovered videos were evaluated with
various semantic and pixel-level metrics. We achieved an average accuracy of
85% in semantic classification tasks and 0.19 in structural similarity index
(SSIM), outperforming the previous state-of-the-art by 45%. We also show that
our model is biologically plausible and interpretable, reflecting established
physiological processes.
Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields
Authors: name: Jingbo Zhang, name: Xiaoyu Li, name: Ziyu Wan, name: Can Wang, name: Jing Liao
Published: 2023-05-19 10:58:04

Text-driven 3D scene generation is widely applicable to video gaming, film
industry, and metaverse applications that have a large demand for 3D scenes.
However, existing text-to-3D generation methods are limited to producing 3D
objects with simple geometries and dreamlike styles that lack realism. In this
work, we present Text2NeRF, which is able to generate a wide range of 3D scenes
with complicated geometric structures and high-fidelity textures purely from a
text prompt. To this end, we adopt NeRF as the 3D representation and leverage a
pre-trained text-to-image diffusion model to constrain the 3D reconstruction of
the NeRF to reflect the scene description. Specifically, we employ the
diffusion model to infer the text-related image as the content prior and use a
monocular depth estimation method to offer the geometric prior. Both content
and geometric priors are utilized to update the NeRF model. To guarantee
textured and geometric consistency between different views, we introduce a
progressive scene inpainting and updating strategy for novel view synthesis of
the scene. Our method requires no additional training data but only a natural
language description of the scene as the input. Extensive experiments
demonstrate that our Text2NeRF outperforms existing methods in producing
photo-realistic, multi-view consistent, and diverse 3D scenes from a variety of
natural language prompts.
RoomDreamer: Text-Driven 3D Indoor Scene Synthesis with Coherent
  Geometry and Texture
Authors: name: Liangchen Song, name: Liangliang Cao, name: Hongyu Xu, name: Kai Kang, name: Feng Tang, name: Junsong Yuan, name: Yang Zhao
Published: 2023-05-18 22:57:57

The techniques for 3D indoor scene capturing are widely used, but the meshes
produced leave much to be desired. In this paper, we propose "RoomDreamer",
which leverages powerful natural language to synthesize a new room with a
different style. Unlike existing image synthesis methods, our work addresses
the challenge of synthesizing both geometry and texture aligned to the input
scene structure and prompt simultaneously. The key insight is that a scene
should be treated as a whole, taking into account both scene texture and
geometry. The proposed framework consists of two significant components:
Geometry Guided Diffusion and Mesh Optimization. Geometry Guided Diffusion for
3D Scene guarantees the consistency of the scene style by applying the 2D prior
to the entire scene simultaneously. Mesh Optimization improves the geometry and
texture jointly and eliminates the artifacts in the scanned scene. To validate
the proposed method, real indoor scenes scanned with smartphones are used for
extensive experiments, through which the effectiveness of our method is
demonstrated.
Going Denser with Open-Vocabulary Part Segmentation
Authors: name: Peize Sun, name: Shoufa Chen, name: Chenchen Zhu, name: Fanyi Xiao, name: Ping Luo, name: Saining Xie, name: Zhicheng Yan
Published: 2023-05-18 17:59:10

Object detection has been expanded from a limited number of categories to
open vocabulary. Moving forward, a complete intelligent vision system requires
understanding more fine-grained object descriptions, object parts. In this
paper, we propose a detector with the ability to predict both open-vocabulary
objects and their part segmentation. This ability comes from two designs.
First, we train the detector on the joint of part-level, object-level and
image-level data to build the multi-granularity alignment between language and
image. Second, we parse the novel object into its parts by its dense semantic
correspondence with the base object. These two designs enable the detector to
largely benefit from various data sources and foundation models. In
open-vocabulary part segmentation experiments, our method outperforms the
baseline by 3.3sim7.3 mAP in cross-dataset generalization on PartImageNet,
and improves the baseline by 7.3 novel AP_{50} in cross-category
generalization on Pascal Part. Finally, we train a detector that generalizes to
a wide range of part segmentation datasets while achieving better performance
than dataset-specific training.
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal
  Conversational Abilities
Authors: name: Dong Zhang, name: Shimin Li, name: Xin Zhang, name: Jun Zhan, name: Pengyu Wang, name: Yaqian Zhou, name: Xipeng Qiu
Published: 2023-05-18 14:23:25

Multi-modal large language models are regarded as a crucial step towards
Artificial General Intelligence (AGI) and have garnered significant interest
with the emergence of ChatGPT. However, current speech-language models
typically adopt the cascade paradigm, preventing inter-modal knowledge
transfer. In this paper, we propose SpeechGPT, a large language model with
intrinsic cross-modal conversational abilities, capable of perceiving and
generating multi-model content. With discrete speech representations, we first
construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.
Additionally, we employ a three-stage training strategy that includes
modality-adaptation pre-training, cross-modal instruction fine-tuning, and
chain-of-modality instruction fine-tuning. The experimental results demonstrate
that SpeechGPT has an impressive capacity to follow multi-modal human
instructions and highlight the potential of handling multiple modalities with
one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
Drag Your GAN: Interactive Point-based Manipulation on the Generative
  Image Manifold
Authors: name: Xingang Pan, name: Ayush Tewari, name: Thomas Leimkühler, name: Lingjie Liu, name: Abhimitra Meka, name: Christian Theobalt
Published: 2023-05-18 13:41:25

Synthesizing visual content that meets users' needs often requires flexible
and precise controllability of the pose, shape, expression, and layout of the
generated objects. Existing approaches gain controllability of generative
adversarial networks (GANs) via manually annotated training data or a prior 3D
model, which often lack flexibility, precision, and generality. In this work,
we study a powerful yet much less explored way of controlling GANs, that is, to
"drag" any points of the image to precisely reach target points in a
user-interactive manner, as shown in Fig.1. To achieve this, we propose
DragGAN, which consists of two main components: 1) a feature-based motion
supervision that drives the handle point to move towards the target position,
and 2) a new point tracking approach that leverages the discriminative
generator features to keep localizing the position of the handle points.
Through DragGAN, anyone can deform an image with precise control over where
pixels go, thus manipulating the pose, shape, expression, and layout of diverse
categories such as animals, cars, humans, landscapes, etc. As these
manipulations are performed on the learned generative image manifold of a GAN,
they tend to produce realistic outputs even for challenging scenarios such as
hallucinating occluded content and deforming shapes that consistently follow
the object's rigidity. Both qualitative and quantitative comparisons
demonstrate the advantage of DragGAN over prior approaches in the tasks of
image manipulation and point tracking. We also showcase the manipulation of
real images through GAN inversion.