Orionnnn/hub-stats huggingface dataset:
GaussianSR: 3D Gaussian Super-Resolution with 2D Diffusion Priors
Authors: name: Xiqian Yu, name: Hanxin Zhu, name: Tianyu He, name: Zhibo Chen
Published: 2024-06-14 15:19:21

Achieving high-resolution novel view synthesis (HRNVS) from low-resolution
input views is a challenging task due to the lack of high-resolution data.
Previous methods optimize high-resolution Neural Radiance Field (NeRF) from
low-resolution input views but suffer from slow rendering speed. In this work,
we base our method on 3D Gaussian Splatting (3DGS) due to its capability of
producing high-quality images at a faster rendering speed. To alleviate the
shortage of data for higher-resolution synthesis, we propose to leverage
off-the-shelf 2D diffusion priors by distilling the 2D knowledge into 3D with
Score Distillation Sampling (SDS). Nevertheless, applying SDS directly to
Gaussian-based 3D super-resolution leads to undesirable and redundant 3D
Gaussian primitives, due to the randomness brought by generative priors. To
mitigate this issue, we introduce two simple yet effective techniques to reduce
stochastic disturbances introduced by SDS. Specifically, we 1) shrink the range
of diffusion timestep in SDS with an annealing strategy; 2) randomly discard
redundant Gaussian primitives during densification. Extensive experiments have
demonstrated that our proposed GaussainSR can attain high-quality results for
HRNVS with only low-resolution inputs on both synthetic and real-world
datasets. Project page: https://chchnii.github.io/GaussianSR/
GEB-1.3B: Open Lightweight Large Language Model
Authors: name: Jie Wu, name: Yufeng Zhu, name: Lei Shen, name: Xuqing Lu
Published: 2024-06-14 10:15:49

Recently developed large language models (LLMs) such as ChatGPT, Claude, and
Llama have demonstrated impressive abilities, and even surpass human-level
performance in several tasks. Despite their success, the resource-intensive
demands of these models, requiring significant computational power for both
training and inference, limit their deployment to high-performance servers.
Additionally, the extensive calculation requirements of the models often lead
to increased latency in response times. With the increasing need for LLMs to
operate efficiently on CPUs, research about lightweight models that are
optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a
lightweight LLM trained on 550 billion tokens in both Chinese and English
languages. We employ novel training techniques, including ROPE,
Group-Query-Attention, and FlashAttention-2, to accelerate training while
maintaining model performance. Additionally, we fine-tune the model using 10
million samples of instruction data to enhance alignment. GEB-1.3B exhibits
outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,
outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.
Notably, the FP32 version of GEB-1.3B achieves commendable inference times on
CPUs, with ongoing efforts to further enhance speed through advanced
quantization techniques. The release of GEB-1.3B as an open-source model marks
a significant contribution to the development of lightweight LLMs, promising to
foster further research and innovation in the field.
MaskLID: Code-Switching Language Identification through Iterative
  Masking
Authors: name: Amir Hossein Kargaran, name: François Yvon, name: Hinrich Schütze
Published: 2024-06-10 13:44:29

We present MaskLID, a simple, yet effective, code-switching (CS) language
identification (LID) method. MaskLID does not require any training and is
designed to complement current high-performance sentence-level LIDs.
Sentence-level LIDs are classifiers trained on monolingual texts to provide
single labels, typically using a softmax layer to turn scores into
probabilities. However, in cases where a sentence is composed in both L1 and L2
languages, the LID classifier often only returns the dominant label L1. To
address this limitation, MaskLID employs a strategy to mask text features
associated with L1, allowing the LID to classify the text as L2 in the next
round. This method uses the LID itself to identify the features that require
masking and does not rely on any external resource. In this work, we explore
the use of MaskLID for two open-source LIDs (GlotLID and OpenLID), that are
both based on the FastText architecture. Code and demo are available at
https://github.com/cisnlp/MaskLID.
XLand-100B: A Large-Scale Multi-Task Dataset for In-Context
  Reinforcement Learning
Authors: name: Alexander Nikulin, name: Ilya Zisman, name: Alexey Zemtsov, name: Viacheslav Sinii, name: Vladislav Kurenkov, name: Sergey Kolesnikov
Published: 2024-06-13 10:04:17

Following the success of the in-context learning paradigm in large-scale
language and computer vision models, the recently emerging field of in-context
reinforcement learning is experiencing a rapid growth. However, its development
has been held back by the lack of challenging benchmarks, as all the
experiments have been carried out in simple environments and on small-scale
datasets. We present XLand-100B, a large-scale dataset for in-context
reinforcement learning based on the XLand-MiniGrid environment, as a first step
to alleviate this problem. It contains complete learning histories for nearly
30,000 different tasks, covering 100B transitions and 2.5B episodes. It
took 50,000 GPU hours to collect the dataset, which is beyond the reach of
most academic labs. Along with the dataset, we provide the utilities to
reproduce or expand it even further. With this substantial effort, we aim to
democratize research in the rapidly growing field of in-context reinforcement
learning and provide a solid foundation for further scaling. The code is
open-source and available under Apache 2.0 licence at
https://github.com/dunno-lab/xland-minigrid-datasets.
ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via
  Chart-to-Code Generation
Authors: name: Chufan Shi, name: Cheng Yang, name: Yaxin Liu, name: Bo Shui, name: Junjie Wang, name: Mohan Jing, name: Linran Xu, name: Xinyu Zhu, name: Siheng Li, name: Yuxiang Zhang, name: Gongye Liu, name: Xiaomei Nie, name: Deng Cai, name: Yujiu Yang
Published: 2024-06-14 12:10:51

We introduce a new benchmark, ChartMimic, aimed at assessing the
visually-grounded code generation capabilities of large multimodal models
(LMMs). ChartMimic utilizes information-intensive visual charts and textual
instructions as inputs, requiring LMMs to generate the corresponding code for
chart rendering. ChartMimic includes 1,000 human-curated (figure, instruction,
code) triplets, which represent the authentic chart use cases found in
scientific papers across various domains(e.g., Physics, Computer Science,
Economics, etc). These charts span 18 regular types and 4 advanced types,
diversifying into 191 subcategories. Furthermore, we propose multi-level
evaluation metrics to provide an automatic and thorough assessment of the
output code and the rendered charts. Unlike existing code generation
benchmarks, ChartMimic places emphasis on evaluating LMMs' capacity to
harmonize a blend of cognitive capabilities, encompassing visual understanding,
code generation, and cross-modal reasoning. The evaluation of 3 proprietary
models and 11 open-weight models highlights the substantial challenges posed by
ChartMimic. Even the advanced GPT-4V, Claude-3-opus only achieve an average
score of 73.2 and 53.7, respectively, indicating significant room for
improvement. We anticipate that ChartMimic will inspire the development of
LMMs, advancing the pursuit of artificial general intelligence.
Designing a Dashboard for Transparency and Control of Conversational AI
Authors: name: Yida Chen, name: Aoyu Wu, name: Trevor DePodesta, name: Catherine Yeh, name: Kenneth Li, name: Nicholas Castillo Marin, name: Oam Patel, name: Jan Riecke, name: Shivam Raval, name: Olivia Seow, name: Martin Wattenberg, name: Fernanda Viégas
Published: 2024-06-12 05:20:16

Conversational LLMs function as black box systems, leaving users guessing
about why they see the output they do. This lack of transparency is potentially
problematic, especially given concerns around bias and truthfulness. To address
this issue, we present an end-to-end prototype-connecting interpretability
techniques with user experience design-that seeks to make chatbots more
transparent. We begin by showing evidence that a prominent open-source LLM has
a "user model": examining the internal state of the system, we can extract data
related to a user's age, gender, educational level, and socioeconomic status.
Next, we describe the design of a dashboard that accompanies the chatbot
interface, displaying this user model in real time. The dashboard can also be
used to control the user model and the system's behavior. Finally, we discuss a
study in which users conversed with the instrumented system. Our results
suggest that users appreciate seeing internal states, which helped them expose
biased behavior and increased their sense of control. Participants also made
valuable suggestions that point to future directions for both design and
machine learning research. The project page and video demo of our TalkTuner
system are available at https://bit.ly/talktuner-project-page
Rethinking Human Evaluation Protocol for Text-to-Video Models: Enhancing
  Reliability,Reproducibility, and Practicality
Authors: name: Tianle Zhang, name: Langtian Ma, name: Yuchen Yan, name: Yuchen Zhang, name: Kai Wang, name: Yue Yang, name: Ziyao Guo, name: Wenqi Shao, name: Yang You, name: Yu Qiao, name: Ping Luo, name: Kaipeng Zhang
Published: 2024-06-13 06:09:22

Recent text-to-video (T2V) technology advancements, as demonstrated by models
such as Gen2, Pika, and Sora, have significantly broadened its applicability
and popularity. Despite these strides, evaluating these models poses
substantial challenges. Primarily, due to the limitations inherent in automatic
metrics, manual evaluation is often considered a superior method for assessing
T2V generation. However, existing manual evaluation protocols face
reproducibility, reliability, and practicality issues. To address these
challenges, this paper introduces the Text-to-Video Human Evaluation (T2VHE)
protocol, a comprehensive and standardized protocol for T2V models. The T2VHE
protocol includes well-defined metrics, thorough annotator training, and an
effective dynamic evaluation module. Experimental results demonstrate that this
protocol not only ensures high-quality annotations but can also reduce
evaluation costs by nearly 50%. We will open-source the entire setup of the
T2VHE protocol, including the complete protocol workflow, the dynamic
evaluation component details, and the annotation interface code. This will help
communities establish more sophisticated human assessment protocols.
GUI Odyssey: A Comprehensive Dataset for Cross-App GUI Navigation on
  Mobile Devices
Authors: name: Quanfeng Lu, name: Wenqi Shao, name: Zitao Liu, name: Fanqing Meng, name: Boxuan Li, name: Botong Chen, name: Siyuan Huang, name: Kaipeng Zhang, name: Yu Qiao, name: Ping Luo
Published: 2024-06-12 17:44:26

Smartphone users often navigate across multiple applications (apps) to
complete tasks such as sharing content between social media platforms.
Autonomous Graphical User Interface (GUI) navigation agents can enhance user
experience in communication, entertainment, and productivity by streamlining
workflows and reducing manual intervention. However, prior GUI agents often
trained with datasets comprising simple tasks that can be completed within a
single app, leading to poor performance in cross-app navigation. To address
this problem, we introduce GUI Odyssey, a comprehensive dataset for training
and evaluating cross-app navigation agents. GUI Odyssey consists of 7,735
episodes from 6 mobile devices, spanning 6 types of cross-app tasks, 201 apps,
and 1.4K app combos. Leveraging GUI Odyssey, we developed OdysseyAgent, a
multimodal cross-app navigation agent by fine-tuning the Qwen-VL model with a
history resampling module. Extensive experiments demonstrate OdysseyAgent's
superior accuracy compared to existing models. For instance, OdysseyAgent
surpasses fine-tuned Qwen-VL and zero-shot GPT-4V by 1.44\% and 55.49\%
in-domain accuracy, and 2.29\% and 48.14\% out-of-domain accuracy on average.
The dataset and code will be released in
https://github.com/OpenGVLab/GUI-Odyssey.
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images
  Interleaved with Text
Authors: name: Qingyun Li, name: Zhe Chen, name: Weiyun Wang, name: Wenhai Wang, name: Shenglong Ye, name: Zhenjiang Jin, name: Guanzhou Chen, name: Yinan He, name: Zhangwei Gao, name: Erfei Cui, name: Jiashuo Yu, name: Hao Tian, name: Jiasheng Zhou, name: Chao Xu, name: Bin Wang, name: Xingjian Wei, name: Wei Li, name: Wenjian Zhang, name: Bo Zhang, name: Pinlong Cai, name: Licheng Wen, name: Xiangchao Yan, name: Zhenxiang Li, name: Pei Chu, name: Yi Wang, name: Min Dou, name: Changyao Tian, name: Xizhou Zhu, name: Lewei Lu, name: Yushi Chen, name: Junjun He, name: Zhongying Tu, name: Tong Lu, name: Yali Wang, name: Limin Wang, name: Dahua Lin, name: Yu Qiao, name: Botian Shi, name: Conghui He, name: Jifeng Dai
Published: 2024-06-12 17:01:04

Image-text interleaved data, consisting of multiple images and texts arranged
in a natural document format, aligns with the presentation paradigm of internet
data and closely resembles human reading habits. Recent studies have shown that
such data aids multimodal in-context learning and maintains the capabilities of
large language models during multimodal fine-tuning. However, the limited scale
and diversity of current image-text interleaved data restrict the development
of multimodal large language models. In this paper, we introduce OmniCorpus, a
10 billion-scale image-text interleaved dataset. Using an efficient data
engine, we filter and extract large-scale high-quality documents, which contain
8.6 billion images and 1,696 billion text tokens. Compared to counterparts
(e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while
maintaining good data quality; 2) features more diverse sources, including both
English and non-English websites as well as video-centric websites; 3) is more
flexible, easily degradable from an image-text interleaved format to pure text
corpus and image-text pairs. Through comprehensive analysis and experiments, we
validate the quality, usability, and effectiveness of the proposed dataset. We
hope this could provide a solid data foundation for future multimodal model
research. Code and data are released at
https://github.com/OpenGVLab/OmniCorpus.
Needle In A Multimodal Haystack
Authors: name: Weiyun Wang, name: Shuibo Zhang, name: Yiming Ren, name: Yuchen Duan, name: Tiantong Li, name: Shuo Liu, name: Mengkang Hu, name: Zhe Chen, name: Kaipeng Zhang, name: Lewei Lu, name: Xizhou Zhu, name: Ping Luo, name: Yu Qiao, name: Jifeng Dai, name: Wenqi Shao, name: Wenhai Wang
Published: 2024-06-11 13:09:16

With the rapid advancement of multimodal large language models (MLLMs), their
evaluation has become increasingly comprehensive. However, understanding long
multimodal content, as a foundational ability for real-world applications,
remains underexplored. In this work, we present Needle In A Multimodal Haystack
(MM-NIAH), the first benchmark specifically designed to systematically evaluate
the capability of existing MLLMs to comprehend long multimodal documents. Our
benchmark includes three types of evaluation tasks: multimodal retrieval,
counting, and reasoning. In each task, the model is required to answer the
questions according to different key information scattered throughout the given
multimodal document. Evaluating the leading MLLMs on MM-NIAH, we observe that
existing models still have significant room for improvement on these tasks,
especially on vision-centric evaluation. We hope this work can provide a
platform for further research on long multimodal document comprehension and
contribute to the advancement of MLLMs. Code and benchmark are released at
https://github.com/OpenGVLab/MM-NIAH.
Decoding the Diversity: A Review of the Indic AI Research Landscape
Authors: name: Sankalp KJ, name: Vinija Jain, name: Sreyoshi Bhaduri, name: Tamoghna Roy, name: Aman Chadha
Published: 2024-06-13 19:55:20

This review paper provides a comprehensive overview of large language model
(LLM) research directions within Indic languages. Indic languages are those
spoken in the Indian subcontinent, including India, Pakistan, Bangladesh, Sri
Lanka, Nepal, and Bhutan, among others. These languages have a rich cultural
and linguistic heritage and are spoken by over 1.5 billion people worldwide.
With the tremendous market potential and growing demand for natural language
processing (NLP) based applications in diverse languages, generative
applications for Indic languages pose unique challenges and opportunities for
research. Our paper deep dives into the recent advancements in Indic generative
modeling, contributing with a taxonomy of research directions, tabulating 84
recent publications. Research directions surveyed in this paper include LLM
development, fine-tuning existing LLMs, development of corpora, benchmarking
and evaluation, as well as publications around specific techniques, tools, and
applications. We found that researchers across the publications emphasize the
challenges associated with limited data availability, lack of standardization,
and the peculiar linguistic complexities of Indic languages. This work aims to
serve as a valuable resource for researchers and practitioners working in the
field of NLP, particularly those focused on Indic languages, and contributes to
the development of more accurate and efficient LLM applications for these
languages.
Glyph-ByT5-v2: A Strong Aesthetic Baseline for Accurate Multilingual
  Visual Text Rendering
Authors: name: Zeyu Liu, name: Weicong Liang, name: Yiming Zhao, name: Bohan Chen, name: Ji Li, name: Yuhui Yuan
Published: 2024-06-14 17:44:09

Recently, Glyph-ByT5 has achieved highly accurate visual text rendering
performance in graphic design images. However, it still focuses solely on
English and performs relatively poorly in terms of visual appeal. In this work,
we address these two fundamental limitations by presenting Glyph-ByT5-v2 and
Glyph-SDXL-v2, which not only support accurate visual text rendering for 10
different languages but also achieve much better aesthetic quality. To achieve
this, we make the following contributions: (i) creating a high-quality
multilingual glyph-text and graphic design dataset consisting of more than 1
million glyph-text pairs and 10 million graphic design image-text pairs
covering nine other languages, (ii) building a multilingual visual paragraph
benchmark consisting of 1,000 prompts, with 100 for each language, to assess
multilingual visual spelling accuracy, and (iii) leveraging the latest
step-aware preference learning approach to enhance the visual aesthetic
quality. With the combination of these techniques, we deliver a powerful
customized multilingual text encoder, Glyph-ByT5-v2, and a strong aesthetic
graphic generation model, Glyph-SDXL-v2, that can support accurate spelling in
10 different languages. We perceive our work as a significant advancement,
considering that the latest DALL-E3 and Ideogram 1.0 still struggle with the
multilingual visual text rendering task.
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Authors: name: Roman Bachmann, name: Oğuzhan Fatih Kar, name: David Mizrahi, name: Ali Garjani, name: Mingfei Gao, name: David Griffiths, name: Jiaming Hu, name: Afshin Dehghan, name: Amir Zamir
Published: 2024-06-13 17:59:42

Current multimodal and multitask foundation models like 4M or UnifiedIO show
promising results, but in practice their out-of-the-box abilities to accept
diverse inputs and perform diverse tasks are limited by the (usually rather
small) number of modalities and tasks they are trained on. In this paper, we
expand upon the capabilities of them by training a single model on tens of
highly diverse modalities and by performing co-training on large-scale
multimodal datasets and text corpora. This includes training on several
semantic and geometric modalities, feature maps from recent state of the art
models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM
and 4DHumans, and a range of new modalities that allow for novel ways to
interact with the model and steer the generation, for example image metadata or
color palettes. A crucial step in this process is performing discrete
tokenization on various modalities, whether they are image-like, neural network
feature maps, vectors, structured data like instance segmentation or human
poses, or data that can be represented as text. Through this, we expand on the
out-of-the-box capabilities of multimodal models and specifically show the
possibility of training one model to solve at least 3x more tasks/modalities
than existing ones and doing so without a loss in performance. This enables
more fine-grained and controllable multimodal generation capabilities and
allows us to study the distillation of models trained on diverse data and
objectives into a unified model. We successfully scale the training to a three
billion parameter model using tens of modalities and different datasets. The
resulting models and training code are open sourced at 4m.epfl.ch.
Understanding Hallucinations in Diffusion Models through Mode
  Interpolation
Authors: name: Sumukh K Aithal, name: Pratyush Maini, name: Zachary C. Lipton, name: J. Zico Kolter
Published: 2024-06-13 17:43:41

Colloquially speaking, image generation models based upon diffusion processes
are frequently said to exhibit "hallucinations," samples that could never occur
in the training data. But where do such hallucinations come from? In this
paper, we study a particular failure mode in diffusion models, which we term
mode interpolation. Specifically, we find that diffusion models smoothly
"interpolate" between nearby data modes in the training set, to generate
samples that are completely outside the support of the original training
distribution; this phenomenon leads diffusion models to generate artifacts that
never existed in real data (i.e., hallucinations). We systematically study the
reasons for, and the manifestation of this phenomenon. Through experiments on
1D and 2D Gaussians, we show how a discontinuous loss landscape in the
diffusion model's decoder leads to a region where any smooth approximation will
cause such hallucinations. Through experiments on artificial datasets with
various shapes, we show how hallucination leads to the generation of
combinations of shapes that never existed. Finally, we show that diffusion
models in fact know when they go out of support and hallucinate. This is
captured by the high variance in the trajectory of the generated sample towards
the final few backward sampling process. Using a simple metric to capture this
variance, we can remove over 95% of hallucinations at generation time while
retaining 96% of in-support samples. We conclude our exploration by showing the
implications of such hallucination (and its removal) on the collapse (and
stabilization) of recursive training on synthetic data with experiments on
MNIST and 2D Gaussians dataset. We release our code at
https://github.com/locuslab/diffusion-model-hallucination.