Orionnnn/hub-stats huggingface dataset:
ViTAR: Vision Transformer with Any Resolution
Authors: name: Qihang Fan, name: Quanzeng You, name: Xiaotian Han, name: Yongfei Liu, name: Yunzhe Tao, name: Huaibo Huang, name: Ran He, name: Hongxia Yang
Published: 2024-03-27 08:53:13

his paper tackles a significant challenge faced by Vision Transformers
(ViTs): their constrained scalability across different image resolutions.
Typically, ViTs experience a performance decline when processing resolutions
different from those seen during training. Our work introduces two key
innovations to address this issue. Firstly, we propose a novel module for
dynamic resolution adjustment, designed with a single Transformer block,
specifically to achieve highly efficient incremental token integration.
Secondly, we introduce fuzzy positional encoding in the Vision Transformer to
provide consistent positional awareness across multiple resolutions, thereby
preventing overfitting to any single training resolution. Our resulting model,
ViTAR (Vision Transformer with Any Resolution), demonstrates impressive
adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and
80.4\% accuracy at a 4032x4032 resolution, all while reducing computational
costs. ViTAR also shows strong performance in downstream tasks such as instance
and semantic segmentation and can easily combined with self-supervised learning
techniques like Masked AutoEncoder. Our work provides a cost-effective solution
for enhancing the resolution scalability of ViTs, paving the way for more
versatile and efficient high-resolution image processing.
BioMedLM: A 2.7B Parameter Language Model Trained On Biomedical Text
Authors: name: Elliot Bolton, name: Abhinav Venigalla, name: Michihiro Yasunaga, name: David Hall, name: Betty Xiong, name: Tony Lee, name: Roxana Daneshjou, name: Jonathan Frankle, name: Percy Liang, name: Michael Carbin, name: Christopher D. Manning
Published: 2024-03-27 10:18:21

Models such as GPT-4 and Med-PaLM 2 have demonstrated impressive performance
on a wide variety of biomedical NLP tasks. However, these models have hundreds
of billions of parameters, are computationally expensive to run, require users
to send their input data over the internet, and are trained on unknown data
sources. Can smaller, more targeted models compete? To address this question,
we build and release BioMedLM, a 2.7 billion parameter GPT-style autoregressive
model trained exclusively on PubMed abstracts and full articles. When
fine-tuned, BioMedLM can produce strong multiple-choice biomedical
question-answering results competitive with much larger models, such as
achieving a score of 57.3% on MedMCQA (dev) and 69.0% on the MMLU Medical
Genetics exam. BioMedLM can also be fine-tuned to produce useful answers to
patient questions on medical topics. This demonstrates that smaller models can
potentially serve as transparent, privacy-preserving, economical and
environmentally friendly foundations for particular NLP applications, such as
in biomedicine. The model is available on the Hugging Face Hub:
https://huggingface.co/stanford-crfm/BioMedLM.
Mini-Gemini: Mining the Potential of Multi-modality Vision Language
  Models
Authors: name: Yanwei Li, name: Yuechen Zhang, name: Chengyao Wang, name: Zhisheng Zhong, name: Yixin Chen, name: Ruihang Chu, name: Shaoteng Liu, name: Jiaya Jia
Published: 2024-03-27 17:59:04

In this work, we introduce Mini-Gemini, a simple and effective framework
enhancing multi-modality Vision Language Models (VLMs). Despite the
advancements in VLMs facilitating basic visual dialog and reasoning, a
performance gap persists compared to advanced models like GPT-4 and Gemini. We
try to narrow the gap by mining the potential of VLMs for better performance
and any-to-any workflow from three aspects, i.e., high-resolution visual
tokens, high-quality data, and VLM-guided generation. To enhance visual tokens,
we propose to utilize an additional visual encoder for high-resolution
refinement without increasing the visual token count. We further construct a
high-quality dataset that promotes precise image comprehension and
reasoning-based generation, expanding the operational scope of current VLMs. In
general, Mini-Gemini further mines the potential of VLMs and empowers current
frameworks with image understanding, reasoning, and generation simultaneously.
Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs)
from 2B to 34B. It is demonstrated to achieve leading performance in several
zero-shot benchmarks and even surpasses the developed private models. Code and
models are available at https://github.com/dvlab-research/MiniGemini.
Long-form factuality in large language models
Authors: name: Jerry Wei, name: Chengrun Yang, name: Xinying Song, name: Yifeng Lu, name: Nathan Hu, name: Dustin Tran, name: Daiyi Peng, name: Ruibo Liu, name: Da Huang, name: Cosmo Du, name: Quoc V. Le
Published: 2024-03-27 17:48:55

Large language models (LLMs) often generate content that contains factual
errors when responding to fact-seeking prompts on open-ended topics. To
benchmark a model's long-form factuality in open domains, we first use GPT-4 to
generate LongFact, a prompt set comprising thousands of questions spanning 38
topics. We then propose that LLM agents can be used as automated evaluators for
long-form factuality through a method which we call Search-Augmented Factuality
Evaluator (SAFE). SAFE utilizes an LLM to break down a long-form response into
a set of individual facts and to evaluate the accuracy of each fact using a
multi-step reasoning process comprising sending search queries to Google Search
and determining whether a fact is supported by the search results. Furthermore,
we propose extending F1 score as an aggregated metric for long-form factuality.
To do so, we balance the percentage of supported facts in a response
(precision) with the percentage of provided facts relative to a hyperparameter
representing a user's preferred response length (recall).
  Empirically, we demonstrate that LLM agents can achieve superhuman rating
performance - on a set of ~16k individual facts, SAFE agrees with crowdsourced
human annotators 72% of the time, and on a random subset of 100 disagreement
cases, SAFE wins 76% of the time. At the same time, SAFE is more than 20 times
cheaper than human annotators. We also benchmark thirteen language models on
LongFact across four model families (Gemini, GPT, Claude, and PaLM-2), finding
that larger language models generally achieve better long-form factuality.
LongFact, SAFE, and all experimental code are available at
https://github.com/google-deepmind/long-form-factuality.
Gamba: Marry Gaussian Splatting with Mamba for single view 3D
  reconstruction
Authors: name: Qiuhong Shen, name: Xuanyu Yi, name: Zike Wu, name: Pan Zhou, name: Hanwang Zhang, name: Shuicheng Yan, name: Xinchao Wang
Published: 2024-03-27 17:40:14

We tackle the challenge of efficiently reconstructing a 3D asset from a
single image with growing demands for automated 3D content creation pipelines.
Previous methods primarily rely on Score Distillation Sampling (SDS) and Neural
Radiance Fields (NeRF). Despite their significant success, these approaches
encounter practical limitations due to lengthy optimization and considerable
memory usage. In this report, we introduce Gamba, an end-to-end amortized 3D
reconstruction model from single-view images, emphasizing two main insights:
(1) 3D representation: leveraging a large number of 3D Gaussians for an
efficient 3D Gaussian splatting process; (2) Backbone design: introducing a
Mamba-based sequential network that facilitates context-dependent reasoning and
linear scalability with the sequence (token) length, accommodating a
substantial number of Gaussians. Gamba incorporates significant advancements in
data preprocessing, regularization design, and training methodologies. We
assessed Gamba against existing optimization-based and feed-forward 3D
generation approaches using the real-world scanned OmniObject3D dataset. Here,
Gamba demonstrates competitive generation capabilities, both qualitatively and
quantitatively, while achieving remarkable speed, approximately 0.6 second on a
single NVIDIA A100 GPU.
AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
Authors: name: Huawei Wei, name: Zejun Yang, name: Zhisheng Wang
Published: 2024-03-26 13:35:02

In this study, we propose AniPortrait, a novel framework for generating
high-quality animation driven by audio and a reference portrait image. Our
methodology is divided into two stages. Initially, we extract 3D intermediate
representations from audio and project them into a sequence of 2D facial
landmarks. Subsequently, we employ a robust diffusion model, coupled with a
motion module, to convert the landmark sequence into photorealistic and
temporally consistent portrait animation. Experimental results demonstrate the
superiority of AniPortrait in terms of facial naturalness, pose diversity, and
visual quality, thereby offering an enhanced perceptual experience. Moreover,
our methodology exhibits considerable potential in terms of flexibility and
controllability, which can be effectively applied in areas such as facial
motion editing or face reenactment. We release code and model weights at
https://github.com/scutzzj/AniPortrait
2D Gaussian Splatting for Geometrically Accurate Radiance Fields
Authors: name: Binbin Huang, name: Zehao Yu, name: Anpei Chen, name: Andreas Geiger, name: Shenghua Gao
Published: 2024-03-26 17:21:24

3D Gaussian Splatting (3DGS) has recently revolutionized radiance field
reconstruction, achieving high quality novel view synthesis and fast rendering
speed without baking. However, 3DGS fails to accurately represent surfaces due
to the multi-view inconsistent nature of 3D Gaussians. We present 2D Gaussian
Splatting (2DGS), a novel approach to model and reconstruct geometrically
accurate radiance fields from multi-view images. Our key idea is to collapse
the 3D volume into a set of 2D oriented planar Gaussian disks. Unlike 3D
Gaussians, 2D Gaussians provide view-consistent geometry while modeling
surfaces intrinsically. To accurately recover thin surfaces and achieve stable
optimization, we introduce a perspective-accurate 2D splatting process
utilizing ray-splat intersection and rasterization. Additionally, we
incorporate depth distortion and normal consistency terms to further enhance
the quality of the reconstructions. We demonstrate that our differentiable
renderer allows for noise-free and detailed geometry reconstruction while
maintaining competitive appearance quality, fast training speed, and real-time
rendering. Our code will be made publicly available.
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D
  Gaussians
Authors: name: Kerui Ren, name: Lihan Jiang, name: Tao Lu, name: Mulin Yu, name: Linning Xu, name: Zhangkai Ni, name: Bo Dai
Published: 2024-03-26 17:39:36

The recent 3D Gaussian splatting (3D-GS) has shown remarkable rendering
fidelity and efficiency compared to NeRF-based neural scene representations.
While demonstrating the potential for real-time rendering, 3D-GS encounters
rendering bottlenecks in large scenes with complex details due to an excessive
number of Gaussian primitives located within the viewing frustum. This
limitation is particularly noticeable in zoom-out views and can lead to
inconsistent rendering speeds in scenes with varying details. Moreover, it
often struggles to capture the corresponding level of details at different
scales with its heuristic density control operation. Inspired by the
Level-of-Detail (LOD) techniques, we introduce Octree-GS, featuring an
LOD-structured 3D Gaussian approach supporting level-of-detail decomposition
for scene representation that contributes to the final rendering results. Our
model dynamically selects the appropriate level from the set of
multi-resolution anchor points, ensuring consistent rendering performance with
adaptive LOD adjustments while maintaining high-fidelity rendering results.
DreamPolisher: Towards High-Quality Text-to-3D Generation via Geometric
  Diffusion
Authors: name: Yuanze Lin, name: Ronald Clark, name: Philip Torr
Published: 2024-03-25 22:34:05

We present DreamPolisher, a novel Gaussian Splatting based method with
geometric guidance, tailored to learn cross-view consistency and intricate
detail from textual descriptions. While recent progress on text-to-3D
generation methods have been promising, prevailing methods often fail to ensure
view-consistency and textural richness. This problem becomes particularly
noticeable for methods that work with text input alone. To address this, we
propose a two-stage Gaussian Splatting based approach that enforces geometric
consistency among views. Initially, a coarse 3D generation undergoes refinement
via geometric optimization. Subsequently, we use a ControlNet driven refiner
coupled with the geometric consistency term to improve both texture fidelity
and overall consistency of the generated 3D asset. Empirical evaluations across
diverse textual prompts spanning various object categories demonstrate the
efficacy of DreamPolisher in generating consistent and realistic 3D objects,
aligning closely with the semantics of the textual instructions.
InternLM2 Technical Report
Authors: name: Zheng Cai, name: Maosong Cao, name: Haojiong Chen, name: Kai Chen, name: Keyu Chen, name: Xin Chen, name: Xun Chen, name: Zehui Chen, name: Zhi Chen, name: Pei Chu, name: Xiaoyi Dong, name: Haodong Duan, name: Qi Fan, name: Zhaoye Fei, name: Yang Gao, name: Jiaye Ge, name: Chenya Gu, name: Yuzhe Gu, name: Tao Gui, name: Aijia Guo, name: Qipeng Guo, name: Conghui He, name: Yingfan Hu, name: Ting Huang, name: Tao Jiang, name: Penglong Jiao, name: Zhenjiang Jin, name: Zhikai Lei, name: Jiaxing Li, name: Jingwen Li, name: Linyang Li, name: Shuaibin Li, name: Wei Li, name: Yining Li, name: Hongwei Liu, name: Jiangning Liu, name: Jiawei Hong, name: Kaiwen Liu, name: Kuikun Liu, name: Xiaoran Liu, name: Chengqi Lv, name: Haijun Lv, name: Kai Lv, name: Li Ma, name: Runyuan Ma, name: Zerun Ma, name: Wenchang Ning, name: Linke Ouyang, name: Jiantao Qiu, name: Yuan Qu, name: Fukai Shang, name: Yunfan Shao, name: Demin Song, name: Zifan Song, name: Zhihao Sui, name: Peng Sun, name: Yu Sun, name: Huanze Tang, name: Bin Wang, name: Guoteng Wang, name: Jiaqi Wang, name: Jiayu Wang, name: Rui Wang, name: Yudong Wang, name: Ziyi Wang, name: Xingjian Wei, name: Qizhen Weng, name: Fan Wu, name: Yingtong Xiong, name: Chao Xu, name: Ruiliang Xu, name: Hang Yan, name: Yirong Yan, name: Xiaogui Yang, name: Haochen Ye, name: Huaiyuan Ying, name: Jia Yu, name: Jing Yu, name: Yuhang Zang, name: Chuyu Zhang, name: Li Zhang, name: Pan Zhang, name: Peng Zhang, name: Ruijie Zhang, name: Shuo Zhang, name: Songyang Zhang, name: Wenjian Zhang, name: Wenwei Zhang, name: Xingcheng Zhang, name: Xinyue Zhang, name: Hui Zhao, name: Qian Zhao, name: Xiaomeng Zhao, name: Fengzhe Zhou, name: Zaida Zhou, name: Jingming Zhuo, name: Yicheng Zou, name: Xipeng Qiu, name: Yu Qiao, name: Dahua Lin
Published: 2024-03-26 00:53:24

The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has
sparked discussions on the advent of Artificial General Intelligence (AGI).
However, replicating such advancements in open-source models has been
challenging. This paper introduces InternLM2, an open-source LLM that
outperforms its predecessors in comprehensive evaluations across 6 dimensions
and 30 benchmarks, long-context modeling, and open-ended subjective evaluations
through innovative pre-training and optimization techniques. The pre-training
process of InternLM2 is meticulously detailed, highlighting the preparation of
diverse data types including text, code, and long-context data. InternLM2
efficiently captures long-term dependencies, initially trained on 4k tokens
before advancing to 32k tokens in pre-training and fine-tuning stages,
exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test.
InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel
Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF)
strategy that addresses conflicting human preferences and reward hacking. By
releasing InternLM2 models in different training stages and model sizes, we
provide the community with insights into the model's evolution.
Improving Text-to-Image Consistency via Automatic Prompt Optimization
Authors: name: Oscar Mañas, name: Pietro Astolfi, name: Melissa Hall, name: Candace Ross, name: Jack Urbanek, name: Adina Williams, name: Aishwarya Agrawal, name: Adriana Romero-Soriano, name: Michal Drozdzal
Published: 2024-03-26 15:42:01

Impressive advances in text-to-image (T2I) generative models have yielded a
plethora of high performing models which are able to generate aesthetically
appealing, photorealistic images. Despite the progress, these models still
struggle to produce images that are consistent with the input prompt,
oftentimes failing to capture object quantities, relations and attributes
properly. Existing solutions to improve prompt-image consistency suffer from
the following challenges: (1) they oftentimes require model fine-tuning, (2)
they only focus on nearby prompt samples, and (3) they are affected by
unfavorable trade-offs among image quality, representation diversity, and
prompt-image consistency. In this paper, we address these challenges and
introduce a T2I optimization-by-prompting framework, OPT2I, which leverages a
large language model (LLM) to improve prompt-image consistency in T2I models.
Our framework starts from a user prompt and iteratively generates revised
prompts with the goal of maximizing a consistency score. Our extensive
validation on two datasets, MSCOCO and PartiPrompts, shows that OPT2I can boost
the initial consistency score by up to 24.9% in terms of DSG score while
preserving the FID and increasing the recall between generated and real data.
Our work paves the way toward building more reliable and robust T2I systems by
harnessing the power of LLMs.
Fully-fused Multi-Layer Perceptrons on Intel Data Center GPUs
Authors: name: Kai Yuan, name: Christoph Bauinger, name: Xiangyi Zhang, name: Pascal Baehr, name: Matthias Kirchhart, name: Darius Dabert, name: Adrien Tousnakhoff, name: Pierre Boudier, name: Michael Paulitsch
Published: 2024-03-26 11:38:39

This paper presents a SYCL implementation of Multi-Layer Perceptrons (MLPs),
which targets and is optimized for the Intel Data Center GPU Max 1550. To
increase the performance, our implementation minimizes the slow global memory
accesses by maximizing the data reuse within the general register file and the
shared local memory by fusing the operations in each layer of the MLP. We show
with a simple roofline model that this results in a significant increase in the
arithmetic intensity, leading to improved performance, especially for
inference. We compare our approach to a similar CUDA implementation for MLPs
and show that our implementation on the Intel Data Center GPU outperforms the
CUDA implementation on Nvidia's H100 GPU by a factor up to 2.84 in inference
and 1.75 in training. The paper also showcases the efficiency of our SYCL
implementation in three significant areas: Image Compression, Neural Radiance
Fields, and Physics-Informed Machine Learning. In all cases, our implementation
outperforms the off-the-shelf Intel Extension for PyTorch (IPEX) implementation
on the same Intel GPU by up to a factor of 30 and the CUDA PyTorch version on
Nvidia's H100 GPU by up to a factor 19. The code can be found at
https://github.com/intel/tiny-dpcpp-nn.
The Unreasonable Ineffectiveness of the Deeper Layers
Authors: name: Andrey Gromov, name: Kushal Tirumala, name: Hassan Shapourian, name: Paolo Glorioso, name: Daniel A. Roberts
Published: 2024-03-26 17:20:04

We empirically study a simple layer-pruning strategy for popular families of
open-weight pretrained LLMs, finding minimal degradation of performance on
different question-answering benchmarks until after a large fraction (up to
half) of the layers are removed. To prune these models, we identify the optimal
block of layers to prune by considering similarity across layers; then, to
"heal" the damage, we perform a small amount of finetuning. In particular, we
use parameter-efficient finetuning (PEFT) methods, specifically quantization
and Low Rank Adapters (QLoRA), such that each of our experiments can be
performed on a single A100 GPU. From a practical perspective, these results
suggest that layer pruning methods can complement other PEFT strategies to
further reduce computational resources of finetuning on the one hand, and can
improve the memory and latency of inference on the other hand. From a
scientific perspective, the robustness of these LLMs to the deletion of layers
implies either that current pretraining methods are not properly leveraging the
parameters in the deeper layers of the network or that the shallow layers play
a critical role in storing knowledge.
TC4D: Trajectory-Conditioned Text-to-4D Generation
Authors: name: Sherwin Bahmani, name: Xian Liu, name: Yifan Wang, name: Ivan Skorokhodov, name: Victor Rong, name: Ziwei Liu, name: Xihui Liu, name: Jeong Joon Park, name: Sergey Tulyakov, name: Gordon Wetzstein, name: Andrea Tagliasacchi, name: David B. Lindell
Published: 2024-03-26 17:55:11

Recent techniques for text-to-4D generation synthesize dynamic 3D scenes
using supervision from pre-trained text-to-video models. However, existing
representations for motion, such as deformation models or time-dependent neural
representations, are limited in the amount of motion they can generate-they
cannot synthesize motion extending far beyond the bounding box used for volume
rendering. The lack of a more flexible motion model contributes to the gap in
realism between 4D generation methods and recent, near-photorealistic video
generation models. Here, we propose TC4D: trajectory-conditioned text-to-4D
generation, which factors motion into global and local components. We represent
the global motion of a scene's bounding box using rigid transformation along a
trajectory parameterized by a spline. We learn local deformations that conform
to the global trajectory using supervision from a text-to-video model. Our
approach enables the synthesis of scenes animated along arbitrary trajectories,
compositional scene generation, and significant improvements to the realism and
amount of generated motion, which we evaluate qualitatively and through a user
study. Video results can be viewed on our website:
https://sherwinbahmani.github.io/tc4d.
RakutenAI-7B: Extending Large Language Models for Japanese
Authors: name: Rakuten Group, name: Aaron Levine, name: Connie Huang, name: Chenguang Wang, name: Eduardo Batista, name: Ewa Szymanska, name: Hongyi Ding, name: Hou Wei Chou, name: Jean-François Pessiot, name: Johanes Effendi, name: Justin Chiu, name: Kai Torben Ohlhus, name: Karan Chopra, name: Keiji Shinzato, name: Koji Murakami, name: Lee Xiong, name: Lei Chen, name: Maki Kubota, name: Maksim Tkachenko, name: Miroku Lee, name: Naoki Takahashi, name: Prathyusha Jwalapuram, name: Ryutaro Tatsushima, name: Saurabh Jain, name: Sunil Kumar Yadav, name: Ting Cai, name: Wei-Te Chen, name: Yandi Xia, name: Yuki Nakayama, name: Yutaka Higashiyama
Published: 2024-03-21 06:56:07

We introduce RakutenAI-7B, a suite of Japanese-oriented large language models
that achieve the best performance on the Japanese LM Harness benchmarks among
the open 7B models. Along with the foundation model, we release instruction-
and chat-tuned models, RakutenAI-7B-instruct and RakutenAI-7B-chat
respectively, under the Apache 2.0 license.