Orionnnn/hub-stats huggingface dataset:
Nougat: Neural Optical Understanding for Academic Documents
Authors: name: Lukas Blecher, name: Guillem Cucurull, name: Thomas Scialom, name: Robert Stojnic
Published: 2023-08-25 15:03:36

Scientific knowledge is predominantly stored in books and scientific
journals, often in the form of PDFs. However, the PDF format leads to a loss of
semantic information, particularly for mathematical expressions. We propose
Nougat (Neural Optical Understanding for Academic Documents), a Visual
Transformer model that performs an Optical Character Recognition (OCR) task for
processing scientific documents into a markup language, and demonstrate the
effectiveness of our model on a new dataset of scientific documents. The
proposed approach offers a promising solution to enhance the accessibility of
scientific knowledge in the digital age, by bridging the gap between
human-readable documents and machine-readable text. We release the models and
code to accelerate future work on scientific text recognition.
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language
  Models
Authors: name: Wenqi Shao, name: Mengzhao Chen, name: Zhaoyang Zhang, name: Peng Xu, name: Lirui Zhao, name: Zhiqian Li, name: Kaipeng Zhang, name: Peng Gao, name: Yu Qiao, name: Ping Luo
Published: 2023-08-25 02:28:35

Large language models (LLMs) have revolutionized natural language processing
tasks. However, their practical deployment is hindered by their immense memory
and computation requirements. Although recent post-training quantization (PTQ)
methods are effective in reducing memory footprint and improving the
computational efficiency of LLM, they hand-craft quantization parameters, which
leads to low performance and fails to deal with extremely low-bit quantization.
To tackle this issue, we introduce an Omnidirectionally calibrated Quantization
(OmniQuant) technique for LLMs, which achieves good performance in diverse
quantization settings while maintaining the computational efficiency of PTQ by
efficiently optimizing various quantization parameters. OmniQuant comprises two
innovative components including Learnable Weight Clipping (LWC) and Learnable
Equivalent Transformation (LET). LWC modulates the extreme values of weights by
optimizing the clipping threshold. Meanwhile, LET tackles activation outliers
by shifting the challenge of quantization from activations to weights through a
learnable equivalent transformation. Operating within a differentiable
framework using block-wise error minimization, OmniQuant can optimize the
quantization process efficiently for both weight-only and weight-activation
quantization. For instance, the LLaMA-2 model family with the size of 7-70B can
be processed with OmniQuant on a single A100-40G GPU within 1-16 hours using
128 samples. Extensive experiments validate OmniQuant's superior performance
across diverse quantization configurations such as W4A4, W6A6, W4A16, W3A16,
and W2A16. Additionally, OmniQuant demonstrates effectiveness in
instruction-tuned models and delivers notable improvements in inference speed
and memory reduction on real devices. Codes and models are available at
https://github.com/OpenGVLab/OmniQuant.
SoTaNa: The Open-Source Software Development Assistant
Authors: name: Ensheng Shi, name: Fengji Zhang, name: Yanlin Wang, name: Bei Chen, name: Lun Du, name: Hongyu Zhang, name: Shi Han, name: Dongmei Zhang, name: Hongbin Sun
Published: 2023-08-25 14:56:21

Software development plays a crucial role in driving innovation and
efficiency across modern societies. To meet the demands of this dynamic field,
there is a growing need for an effective software development assistant.
However, existing large language models represented by ChatGPT suffer from
limited accessibility, including training data and model weights. Although
other large open-source models like LLaMA have shown promise, they still
struggle with understanding human intent. In this paper, we present SoTaNa, an
open-source software development assistant. SoTaNa utilizes ChatGPT to generate
high-quality instruction-based data for the domain of software engineering and
employs a parameter-efficient fine-tuning approach to enhance the open-source
foundation model, LLaMA. We evaluate the effectiveness of  in answering
Stack Overflow questions and demonstrate its capabilities. Additionally, we
discuss its capabilities in code summarization and generation, as well as the
impact of varying the volume of generated data on model performance. Notably,
SoTaNa can run on a single GPU, making it accessible to a broader range of
researchers. Our code, model weights, and data are public at
https://github.com/DeepSoftwareAnalytics/SoTaNa.
SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes
Authors: name: Edith Tretschk, name: Vladislav Golyanik, name: Michael Zollhoefer, name: Aljaz Bozic, name: Christoph Lassner, name: Christian Theobalt
Published: 2023-08-16 09:50:35

Existing methods for the 4D reconstruction of general, non-rigidly deforming
objects focus on novel-view synthesis and neglect correspondences. However,
time consistency enables advanced downstream tasks like 3D editing, motion
analysis, or virtual-asset creation. We propose SceNeRFlow to reconstruct a
general, non-rigid scene in a time-consistent manner. Our dynamic-NeRF method
takes multi-view RGB videos and background images from static cameras with
known camera parameters as input. It then reconstructs the deformations of an
estimated canonical model of the geometry and appearance in an online fashion.
Since this canonical model is time-invariant, we obtain correspondences even
for long-term, long-range motions. We employ neural scene representations to
parametrize the components of our method. Like prior dynamic-NeRF methods, we
use a backwards deformation model. We find non-trivial adaptations of this
model necessary to handle larger motions: We decompose the deformations into a
strongly regularized coarse component and a weakly regularized fine component,
where the coarse component also extends the deformation field into the space
surrounding the object, which enables tracking over time. We show
experimentally that, unlike prior work that only handles small motion, our
method enables the reconstruction of studio-scale motions.
TeCH: Text-guided Reconstruction of Lifelike Clothed Humans
Authors: name: Yangyi Huang, name: Hongwei Yi, name: Yuliang Xiu, name: Tingting Liao, name: Jiaxiang Tang, name: Deng Cai, name: Justus Thies
Published: 2023-08-16 17:59:13

Despite recent research advancements in reconstructing clothed humans from a
single image, accurately restoring the "unseen regions" with high-level details
remains an unsolved challenge that lacks attention. Existing methods often
generate overly smooth back-side surfaces with a blurry texture. But how to
effectively capture all visual attributes of an individual from a single image,
which are sufficient to reconstruct unseen areas (e.g., the back view)?
Motivated by the power of foundation models, TeCH reconstructs the 3D human by
leveraging 1) descriptive text prompts (e.g., garments, colors, hairstyles)
which are automatically generated via a garment parsing model and Visual
Question Answering (VQA), 2) a personalized fine-tuned Text-to-Image diffusion
model (T2I) which learns the "indescribable" appearance. To represent
high-resolution 3D clothed humans at an affordable cost, we propose a hybrid 3D
representation based on DMTet, which consists of an explicit body shape grid
and an implicit distance field. Guided by the descriptive prompts +
personalized T2I diffusion model, the geometry and texture of the 3D humans are
optimized through multi-view Score Distillation Sampling (SDS) and
reconstruction losses based on the original observation. TeCH produces
high-fidelity 3D clothed humans with consistent & delicate texture, and
detailed full-body geometry. Quantitative and qualitative experiments
demonstrate that TeCH outperforms the state-of-the-art methods in terms of
reconstruction accuracy and rendering quality. The code will be publicly
available for research purposes at https://huangyangyi.github.io/tech
Dual-Stream Diffusion Net for Text-to-Video Generation
Authors: name: Binhui Liu, name: Xin Liu, name: Anbo Dai, name: Zhiyong Zeng, name: Zhen Cui, name: Jian Yang
Published: 2023-08-16 12:22:29

With the emerging diffusion models, recently, text-to-video generation has
aroused increasing attention. But an important bottleneck therein is that
generative videos often tend to carry some flickers and artifacts. In this
work, we propose a dual-stream diffusion net (DSDN) to improve the consistency
of content variations in generating videos. In particular, the designed two
diffusion streams, video content and motion branches, could not only run
separately in their private spaces for producing personalized video variations
as well as content, but also be well-aligned between the content and motion
domains through leveraging our designed cross-transformer interaction module,
which would benefit the smoothness of generated videos. Besides, we also
introduce motion decomposer and combiner to faciliate the operation on video
motion. Qualitative and quantitative experiments demonstrate that our method
could produce amazing continuous videos with fewer flickers.
DragNUWA: Fine-grained Control in Video Generation by Integrating Text,
  Image, and Trajectory
Authors: name: Shengming Yin, name: Chenfei Wu, name: Jian Liang, name: Jie Shi, name: Houqiang Li, name: Gong Ming, name: Nan Duan
Published: 2023-08-16 01:43:41

Controllable video generation has gained significant attention in recent
years. However, two main limitations persist: Firstly, most existing works
focus on either text, image, or trajectory-based control, leading to an
inability to achieve fine-grained control in videos. Secondly, trajectory
control research is still in its early stages, with most experiments being
conducted on simple datasets like Human3.6M. This constraint limits the models'
capability to process open-domain images and effectively handle complex curved
trajectories. In this paper, we propose DragNUWA, an open-domain
diffusion-based video generation model. To tackle the issue of insufficient
control granularity in existing works, we simultaneously introduce text, image,
and trajectory information to provide fine-grained control over video content
from semantic, spatial, and temporal perspectives. To resolve the problem of
limited open-domain trajectory control in current research, We propose
trajectory modeling with three aspects: a Trajectory Sampler (TS) to enable
open-domain control of arbitrary trajectories, a Multiscale Fusion (MF) to
control trajectories in different granularities, and an Adaptive Training (AT)
strategy to generate consistent videos following trajectories. Our experiments
validate the effectiveness of DragNUWA, demonstrating its superior performance
in fine-grained control in video generation. The homepage link is
https://www.microsoft.com/en-us/research/project/dragnuwa/
Teach LLMs to Personalize -- An Approach inspired by Writing Education
Authors: name: Cheng Li, name: Mingyang Zhang, name: Qiaozhu Mei, name: Yaqing Wang, name: Spurthi Amba Hombaiah, name: Yi Liang, name: Michael Bendersky
Published: 2023-08-15 18:06:23

Personalized text generation is an emerging research area that has attracted
much attention in recent years. Most studies in this direction focus on a
particular domain by designing bespoke features or models. In this work, we
propose a general approach for personalized text generation using large
language models (LLMs). Inspired by the practice of writing education, we
develop a multistage and multitask framework to teach LLMs for personalized
generation. In writing instruction, the task of writing from sources is often
decomposed into multiple steps that involve finding, evaluating, summarizing,
synthesizing, and integrating information. Analogously, our approach to
personalized text generation consists of multiple stages: retrieval, ranking,
summarization, synthesis, and generation. In addition, we introduce a multitask
setting that helps the model improve its generation ability further, which is
inspired by the observation in education that a student's reading proficiency
and writing ability are often correlated. We evaluate our approach on three
public datasets, each of which covers a different and representative domain.
Our results show significant improvements over a variety of baselines.
Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Authors: name: William Shen, name: Ge Yang, name: Alan Yu, name: Jansen Wong, name: Leslie Pack Kaelbling, name: Phillip Isola
Published: 2023-07-27 17:59:14

Self-supervised and language-supervised image models contain rich knowledge
of the world that is important for generalization. Many robotic tasks, however,
require a detailed understanding of 3D geometry, which is often lacking in 2D
image features. This work bridges this 2D-to-3D gap for robotic manipulation by
leveraging distilled feature fields to combine accurate 3D geometry with rich
semantics from 2D foundation models. We present a few-shot learning method for
6-DOF grasping and placing that harnesses these strong spatial and semantic
priors to achieve in-the-wild generalization to unseen objects. Using features
distilled from a vision-language model, CLIP, we present a way to designate
novel objects for manipulation via free-text natural language, and demonstrate
its ability to generalize to unseen expressions and novel categories of
objects.
CoDeF: Content Deformation Fields for Temporally Consistent Video
  Processing
Authors: name: Hao Ouyang, name: Qiuyu Wang, name: Yuxi Xiao, name: Qingyan Bai, name: Juntao Zhang, name: Kecheng Zheng, name: Xiaowei Zhou, name: Qifeng Chen, name: Yujun Shen
Published: 2023-08-15 17:59:56

We present the content deformation field CoDeF as a new type of video
representation, which consists of a canonical content field aggregating the
static contents in the entire video and a temporal deformation field recording
the transformations from the canonical image (i.e., rendered from the canonical
content field) to each individual frame along the time axis.Given a target
video, these two fields are jointly optimized to reconstruct it through a
carefully tailored rendering pipeline.We advisedly introduce some
regularizations into the optimization process, urging the canonical content
field to inherit semantics (e.g., the object shape) from the video.With such a
design, CoDeF naturally supports lifting image algorithms for video processing,
in the sense that one can apply an image algorithm to the canonical image and
effortlessly propagate the outcomes to the entire video with the aid of the
temporal deformation field.We experimentally show that CoDeF is able to lift
image-to-image translation to video-to-video translation and lift keypoint
detection to keypoint tracking without any training.More importantly, thanks to
our lifting strategy that deploys the algorithms on only one image, we achieve
superior cross-frame consistency in processed videos compared to existing
video-to-video translation approaches, and even manage to track non-rigid
objects like water and smog.Project page can be found at
https://qiuyu96.github.io/CoDeF/.
Relightable and Animatable Neural Avatar from Sparse-View Video
Authors: name: Zhen Xu, name: Sida Peng, name: Chen Geng, name: Linzhan Mou, name: Zihan Yan, name: Jiaming Sun, name: Hujun Bao, name: Xiaowei Zhou
Published: 2023-08-15 17:42:39

This paper tackles the challenge of creating relightable and animatable
neural avatars from sparse-view (or even monocular) videos of dynamic humans
under unknown illumination. Compared to studio environments, this setting is
more practical and accessible but poses an extremely challenging ill-posed
problem. Previous neural human reconstruction methods are able to reconstruct
animatable avatars from sparse views using deformed Signed Distance Fields
(SDF) but cannot recover material parameters for relighting. While
differentiable inverse rendering-based methods have succeeded in material
recovery of static objects, it is not straightforward to extend them to dynamic
humans as it is computationally intensive to compute pixel-surface intersection
and light visibility on deformed SDFs for inverse rendering. To solve this
challenge, we propose a Hierarchical Distance Query (HDQ) algorithm to
approximate the world space distances under arbitrary human poses.
Specifically, we estimate coarse distances based on a parametric human model
and compute fine distances by exploiting the local deformation invariance of
SDF. Based on the HDQ algorithm, we leverage sphere tracing to efficiently
estimate the surface intersection and light visibility. This allows us to
develop the first system to recover animatable and relightable neural avatars
from sparse view (or monocular) inputs. Experiments demonstrate that our
approach is able to produce superior results compared to state-of-the-art
methods. Our code will be released for reproducibility.
Learning to Identify Critical States for Reinforcement Learning from
  Videos
Authors: name: Haozhe Liu, name: Mingchen Zhuge, name: Bing Li, name: Yuhui Wang, name: Francesco Faccio, name: Bernard Ghanem, name: Jürgen Schmidhuber
Published: 2023-08-15 14:21:24

Recent work on deep reinforcement learning (DRL) has pointed out that
algorithmic information about good policies can be extracted from offline data
which lack explicit information about executed actions. For example, videos of
humans or robots may convey a lot of implicit information about rewarding
action sequences, but a DRL machine that wants to profit from watching such
videos must first learn by itself to identify and recognize relevant
states/actions/rewards. Without relying on ground-truth annotations, our new
method called Deep State Identifier learns to predict returns from episodes
encoded as videos. Then it uses a kind of mask-based sensitivity analysis to
extract/identify important critical states. Extensive experiments showcase our
method's potential for understanding and improving agent behavior. The source
code and the generated datasets are available at
https://github.com/AI-Initiative-KAUST/VideoRLCS.
Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with
  Code-based Self-Verification
Authors: name: Aojun Zhou, name: Ke Wang, name: Zimu Lu, name: Weikang Shi, name: Sichun Luo, name: Zipeng Qin, name: Shaoqing Lu, name: Anya Jia, name: Linqi Song, name: Mingjie Zhan, name: Hongsheng Li
Published: 2023-08-15 17:58:45

Recent progress in large language models (LLMs) like GPT-4 and PaLM-2 has
brought significant advancements in addressing math reasoning problems. In
particular, OpenAI's latest version of GPT-4, known as GPT-4 Code Interpreter,
shows remarkable performance on challenging math datasets. In this paper, we
explore the effect of code on enhancing LLMs' reasoning capability by
introducing different constraints on the Code Usage Frequency of GPT-4
Code Interpreter. We found that its success can be largely attributed to its
powerful skills in generating and executing code, evaluating the output of code
execution, and rectifying its solution when receiving unreasonable outputs.
Based on this insight, we propose a novel and effective prompting method,
explicit code-based self-verification~(CSV), to further
boost the mathematical reasoning potential of GPT-4 Code Interpreter. This
method employs a zero-shot prompt on GPT-4 Code Interpreter to encourage it to
use code to self-verify its answers. In instances where the verification state
registers as ``False'', the model shall automatically amend its solution,
analogous to our approach of rectifying errors during a mathematics
examination. Furthermore, we recognize that the states of the verification
result indicate the confidence of a solution, which can improve the
effectiveness of majority voting. With GPT-4 Code Interpreter and CSV, we
achieve an impressive zero-shot accuracy on MATH dataset (53.9\% to
84.3\%).
RAVEN: In-Context Learning with Retrieval Augmented Encoder-Decoder
  Language Models
Authors: name: Jie Huang, name: Wei Ping, name: Peng Xu, name: Mohammad Shoeybi, name: Kevin Chen-Chuan Chang, name: Bryan Catanzaro
Published: 2023-08-15 17:59:18

In this paper, we investigate the in-context learning ability of
retrieval-augmented encoder-decoder language models. We first conduct a
comprehensive analysis of the state-of-the-art ATLAS model and identify its
limitations in in-context learning, primarily due to a mismatch between
pretraining and testing, as well as a restricted context length. To address
these issues, we propose RAVEN, a model that combines retrieval-augmented
masked language modeling and prefix language modeling. We further introduce
Fusion-in-Context Learning to enhance the few-shot performance by enabling the
model to leverage more in-context examples without requiring additional
training or model modifications. Through extensive experiments, we demonstrate
that RAVEN significantly outperforms ATLAS and achieves results comparable to
the most advanced language models in certain scenarios, despite having
substantially fewer parameters. Our work underscores the potential of
retrieval-augmented encoder-decoder language models for in-context learning and
encourages further research in this direction.
Link-Context Learning for Multimodal LLMs
Authors: name: Yan Tai, name: Weichen Fan, name: Zhao Zhang, name: Feng Zhu, name: Rui Zhao, name: Ziwei Liu
Published: 2023-08-15 17:33:24

The ability to learn from context with novel concepts, and deliver
appropriate responses are essential in human conversations. Despite current
Multimodal Large Language Models (MLLMs) and Large Language Models (LLMs) being
trained on mega-scale datasets, recognizing unseen images or understanding
novel concepts in a training-free manner remains a challenge. In-Context
Learning (ICL) explores training-free few-shot learning, where models are
encouraged to ``learn to learn" from limited tasks and generalize to unseen
tasks. In this work, we propose link-context learning (LCL), which emphasizes
"reasoning from cause and effect" to augment the learning capabilities of
MLLMs. LCL goes beyond traditional ICL by explicitly strengthening the causal
relationship between the support set and the query set. By providing
demonstrations with causal links, LCL guides the model to discern not only the
analogy but also the underlying causal associations between data points, which
empowers MLLMs to recognize unseen images and understand novel concepts more
effectively. To facilitate the evaluation of this novel approach, we introduce
the ISEKAI dataset, comprising exclusively of unseen generated image-label
pairs designed for link-context learning. Extensive experiments show that our
LCL-MLLM exhibits strong link-context learning capabilities to novel concepts
over vanilla MLLMs. Code and data will be released at
https://github.com/isekai-portal/Link-Context-Learning.
Text Injection for Capitalization and Turn-Taking Prediction in Speech
  Models
Authors: name: Shaan Bijwadia, name: Shuo-yiin Chang, name: Weiran Wang, name: Zhong Meng, name: Hao Zhang, name: Tara N. Sainath
Published: 2023-08-14 18:28:04

Text injection for automatic speech recognition (ASR), wherein unpaired
text-only data is used to supplement paired audio-text data, has shown
promising improvements for word error rate. This study examines the use of text
injection for auxiliary tasks, which are the non-ASR tasks often performed by
an E2E model. In this work, we use joint end-to-end and internal language model
training (JEIT) as our text injection algorithm to train an ASR model which
performs two auxiliary tasks. The first is capitalization, which is a
de-normalization task. The second is turn-taking prediction, which attempts to
identify whether a user has completed their conversation turn in a digital
assistant interaction. We show results demonstrating that our text injection
method boosts capitalization performance for long-tail data, and improves
turn-taking detection recall.