Orionnnn/hub-stats huggingface dataset:
Sapiens: Foundation for Human Vision Models
Authors: name: Rawal Khirodkar, name: Timur Bagautdinov, name: Julieta Martinez, name: Su Zhaoen, name: Austin James, name: Peter Selednik, name: Stuart Anderson, name: Shunsuke Saito
Published: 2024-08-22 17:37:27

We present Sapiens, a family of models for four fundamental human-centric
vision tasks - 2D pose estimation, body-part segmentation, depth estimation,
and surface normal prediction. Our models natively support 1K high-resolution
inference and are extremely easy to adapt for individual tasks by simply
fine-tuning models pretrained on over 300 million in-the-wild human images. We
observe that, given the same computational budget, self-supervised pretraining
on a curated dataset of human images significantly boosts the performance for a
diverse set of human-centric tasks. The resulting models exhibit remarkable
generalization to in-the-wild data, even when labeled data is scarce or
entirely synthetic. Our simple model design also brings scalability - model
performance across tasks improves as we scale the number of parameters from 0.3
to 2 billion. Sapiens consistently surpasses existing baselines across various
human-centric benchmarks. We achieve significant improvements over the prior
state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1
mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5%
relative angular error.
xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed
  Representations
Authors: name: Can Qin, name: Congying Xia, name: Krithika Ramakrishnan, name: Michael Ryoo, name: Lifu Tu, name: Yihao Feng, name: Manli Shu, name: Honglu Zhou, name: Anas Awadalla, name: Jun Wang, name: Senthil Purushwalkam, name: Le Xue, name: Yingbo Zhou, name: Huan Wang, name: Silvio Savarese, name: Juan Carlos Niebles, name: Zeyuan Chen, name: Ran Xu, name: Caiming Xiong
Published: 2024-08-22 17:55:22

We present xGen-VideoSyn-1, a text-to-video (T2V) generation model capable of
producing realistic scenes from textual descriptions. Building on recent
advancements, such as OpenAI's Sora, we explore the latent diffusion model
(LDM) architecture and introduce a video variational autoencoder (VidVAE).
VidVAE compresses video data both spatially and temporally, significantly
reducing the length of visual tokens and the computational demands associated
with generating long-sequence videos. To further address the computational
costs, we propose a divide-and-merge strategy that maintains temporal
consistency across video segments. Our Diffusion Transformer (DiT) model
incorporates spatial and temporal self-attention layers, enabling robust
generalization across different timeframes and aspect ratios. We have devised a
data processing pipeline from the very beginning and collected over 13M
high-quality video-text pairs. The pipeline includes multiple steps such as
clipping, text detection, motion estimation, aesthetics scoring, and dense
captioning based on our in-house video-LLM model. Training the VidVAE and DiT
models required approximately 40 and 642 H100 days, respectively. Our model
supports over 14-second 720p video generation in an end-to-end way and
demonstrates competitive performance against state-of-the-art T2V models.
Strategist: Learning Strategic Skills by LLMs via Bi-Level Tree Search
Authors: name: Jonathan Light, name: Min Cai, name: Weiqin Chen, name: Guanzhi Wang, name: Xiusi Chen, name: Wei Cheng, name: Yisong Yue, name: Ziniu Hu
Published: 2024-08-20 08:22:04

In this paper, we propose a new method Strategist that utilizes LLMs to
acquire new skills for playing multi-agent games through a self-improvement
process. Our method gathers quality feedback through self-play simulations with
Monte Carlo tree search and LLM-based reflection, which can then be used to
learn high-level strategic skills such as how to evaluate states that guide the
low-level execution.We showcase how our method can be used in both action
planning and dialogue generation in the context of games, achieving good
performance on both tasks. Specifically, we demonstrate that our method can
help train agents with better performance than both traditional reinforcement
learning-based approaches and other LLM-based skill learning approaches in
games including the Game of Pure Strategy (GOPS) and The Resistance: Avalon.
Controllable Text Generation for Large Language Models: A Survey
Authors: name: Xun Liang, name: Hanyu Wang, name: Yezhaohui Wang, name: Shichao Song, name: Jiawei Yang, name: Simin Niu, name: Jie Hu, name: Dan Liu, name: Shunyu Yao, name: Feiyu Xiong, name: Zhiyu Li
Published: 2024-08-22 17:59:04

In Natural Language Processing (NLP), Large Language Models (LLMs) have
demonstrated high text generation quality. However, in real-world applications,
LLMs must meet increasingly complex requirements. Beyond avoiding misleading or
inappropriate content, LLMs are also expected to cater to specific user needs,
such as imitating particular writing styles or generating text with poetic
richness. These varied demands have driven the development of Controllable Text
Generation (CTG) techniques, which ensure that outputs adhere to predefined
control conditions--such as safety, sentiment, thematic consistency, and
linguistic style--while maintaining high standards of helpfulness, fluency, and
diversity.
  This paper systematically reviews the latest advancements in CTG for LLMs,
offering a comprehensive definition of its core concepts and clarifying the
requirements for control conditions and text quality. We categorize CTG tasks
into two primary types: content control and attribute control. The key methods
are discussed, including model retraining, fine-tuning, reinforcement learning,
prompt engineering, latent space manipulation, and decoding-time intervention.
We analyze each method's characteristics, advantages, and limitations,
providing nuanced insights for achieving generation control. Additionally, we
review CTG evaluation methods, summarize its applications across domains, and
address key challenges in current research, including reduced fluency and
practicality. We also propose several appeals, such as placing greater emphasis
on real-world applications in future research. This paper aims to offer
valuable guidance to researchers and developers in the field. Our reference
list and Chinese version are open-sourced at
https://github.com/IAAR-Shanghai/CTGSurvey.
Hermes 3 Technical Report
Authors: name: Ryan Teknium, name: Jeffrey Quesnelle, name: Chen Guang
Published: 2024-08-15 20:17:33

Instruct (or "chat") tuned models have become the primary way in which most
people interact with large language models. As opposed to "base" or
"foundation" models, instruct-tuned models are optimized to respond to
imperative statements. We present Hermes 3, a neutrally-aligned generalist
instruct and tool use model with strong reasoning and creative abilities. Its
largest version, Hermes 3 405B, achieves state of the art performance among
open weight models on several public benchmarks.
Unboxing Occupational Bias: Grounded Debiasing LLMs with U.S. Labor Data
Authors: name: Atmika Gorti, name: Manas Gaur, name: Aman Chadha
Published: 2024-08-20 23:54:26

Large Language Models (LLMs) are prone to inheriting and amplifying societal
biases embedded within their training data, potentially reinforcing harmful
stereotypes related to gender, occupation, and other sensitive categories. This
issue becomes particularly problematic as biased LLMs can have far-reaching
consequences, leading to unfair practices and exacerbating social inequalities
across various domains, such as recruitment, online content moderation, or even
the criminal justice system. Although prior research has focused on detecting
bias in LLMs using specialized datasets designed to highlight intrinsic biases,
there has been a notable lack of investigation into how these findings
correlate with authoritative datasets, such as those from the U.S. National
Bureau of Labor Statistics (NBLS). To address this gap, we conduct empirical
research that evaluates LLMs in a ``bias-out-of-the-box" setting, analyzing how
the generated outputs compare with the distributions found in NBLS data.
Furthermore, we propose a straightforward yet effective debiasing mechanism
that directly incorporates NBLS instances to mitigate bias within LLMs. Our
study spans seven different LLMs, including instructable, base, and
mixture-of-expert models, and reveals significant levels of bias that are often
overlooked by existing bias detection techniques. Importantly, our debiasing
method, which does not rely on external datasets, demonstrates a substantial
reduction in bias scores, highlighting the efficacy of our approach in creating
fairer and more reliable LLMs.
Out-of-Distribution Detection with Attention Head Masking for Multimodal
  Document Classification
Authors: name: Christos Constantinou, name: Georgios Ioannides, name: Aman Chadha, name: Aaron Elkins, name: Edwin Simpson
Published: 2024-08-20 23:30:00

Detecting out-of-distribution (OOD) data is crucial in machine learning
applications to mitigate the risk of model overconfidence, thereby enhancing
the reliability and safety of deployed systems. The majority of existing OOD
detection methods predominantly address uni-modal inputs, such as images or
texts. In the context of multi-modal documents, there is a notable lack of
extensive research on the performance of these methods, which have primarily
been developed with a focus on computer vision tasks. We propose a novel
methodology termed as attention head masking (AHM) for multi-modal OOD tasks in
document classification systems. Our empirical results demonstrate that the
proposed AHM method outperforms all state-of-the-art approaches and
significantly decreases the false positive rate (FPR) compared to existing
solutions up to 7.5\%. This methodology generalizes well to multi-modal data,
such as documents, where visual and textual information are modeled under the
same Transformer architecture. To address the scarcity of high-quality publicly
available document datasets and encourage further research on OOD detection for
documents, we introduce FinanceDocs, a new document AI dataset. Our code and
dataset are publicly available.
Iterative Object Count Optimization for Text-to-image Diffusion Models
Authors: name: Oz Zafar, name: Lior Wolf, name: Idan Schwartz
Published: 2024-08-21 15:51:46

We address a persistent challenge in text-to-image models: accurately
generating a specified number of objects. Current models, which learn from
image-text pairs, inherently struggle with counting, as training data cannot
depict every possible number of objects for any given object. To solve this, we
propose optimizing the generated image based on a counting loss derived from a
counting model that aggregates an object\'s potential. Employing an
out-of-the-box counting model is challenging for two reasons: first, the model
requires a scaling hyperparameter for the potential aggregation that varies
depending on the viewpoint of the objects, and second, classifier guidance
techniques require modified models that operate on noisy intermediate diffusion
steps. To address these challenges, we propose an iterated online training mode
that improves the accuracy of inferred images while altering the text
conditioning embedding and dynamically adjusting hyperparameters. Our method
offers three key advantages: (i) it can consider non-derivable counting
techniques based on detection models, (ii) it is a zero-shot plug-and-play
solution facilitating rapid changes to the counting techniques and image
generation methods, and (iii) the optimized counting token can be reused to
generate accurate images without additional optimization. We evaluate the
generation of various objects and show significant improvements in accuracy.
The project page is available at https://ozzafar.github.io/count_token.
Scaling Cross-Embodied Learning: One Policy for Manipulation,
  Navigation, Locomotion and Aviation
Authors: name: Ria Doshi, name: Homer Walke, name: Oier Mees, name: Sudeep Dasari, name: Sergey Levine
Published: 2024-08-21 17:57:51

Modern machine learning systems rely on large datasets to attain broad
generalization, and this often poses a challenge in robot learning, where each
robotic platform and task might have only a small dataset. By training a single
policy across many different kinds of robots, a robot learning method can
leverage much broader and more diverse datasets, which in turn can lead to
better generalization and robustness. However, training a single policy on
multi-robot data is challenging because robots can have widely varying sensors,
actuators, and control frequencies. We propose CrossFormer, a scalable and
flexible transformer-based policy that can consume data from any embodiment. We
train CrossFormer on the largest and most diverse dataset to date, 900K
trajectories across 20 different robot embodiments. We demonstrate that the
same network weights can control vastly different robots, including single and
dual arm manipulation systems, wheeled robots, quadcopters, and quadrupeds.
Unlike prior work, our model does not require manual alignment of the
observation or action spaces. Extensive experiments in the real world show that
our method matches the performance of specialist policies tailored for each
embodiment, while also significantly outperforming the prior state of the art
in cross-embodiment learning.
Backward-Compatible Aligned Representations via an Orthogonal
  Transformation Layer
Authors: name: Simone Ricci, name: Niccolò Biondi, name: Federico Pernici, name: Alberto Del Bimbo
Published: 2024-08-16 15:05:28

Visual retrieval systems face significant challenges when updating models
with improved representations due to misalignment between the old and new
representations. The costly and resource-intensive backfilling process involves
recalculating feature vectors for images in the gallery set whenever a new
model is introduced. To address this, prior research has explored
backward-compatible training methods that enable direct comparisons between new
and old representations without backfilling. Despite these advancements,
achieving a balance between backward compatibility and the performance of
independently trained models remains an open problem. In this paper, we address
it by expanding the representation space with additional dimensions and
learning an orthogonal transformation to achieve compatibility with old models
and, at the same time, integrate new information. This transformation preserves
the original feature space's geometry, ensuring that our model aligns with
previous versions while also learning new data. Our Orthogonal Compatible
Aligned (OCA) approach eliminates the need for re-indexing during model updates
and ensures that features can be compared directly across different model
updates without additional mapping functions. Experimental results on CIFAR-100
and ImageNet-1k demonstrate that our method not only maintains compatibility
with previous models but also achieves state-of-the-art accuracy, outperforming
several existing methods.
FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive
  Prompt Weighting
Authors: name: Liyao Jiang, name: Negar Hassanpour, name: Mohammad Salameh, name: Mohan Sai Singamsetti, name: Fengyu Sun, name: Wei Lu, name: Di Niu
Published: 2024-08-21 15:30:35

Text-to-image (T2I) diffusion models have demonstrated impressive
capabilities in generating high-quality images given a text prompt. However,
ensuring the prompt-image alignment remains a considerable challenge, i.e.,
generating images that faithfully align with the prompt's semantics. Recent
works attempt to improve the faithfulness by optimizing the latent code, which
potentially could cause the latent code to go out-of-distribution and thus
produce unrealistic images. In this paper, we propose FRAP, a simple, yet
effective approach based on adaptively adjusting the per-token prompt weights
to improve prompt-image alignment and authenticity of the generated images. We
design an online algorithm to adaptively update each token's weight
coefficient, which is achieved by minimizing a unified objective function that
encourages object presence and the binding of object-modifier pairs. Through
extensive evaluations, we show FRAP generates images with significantly higher
prompt-image alignment to prompts from complex datasets, while having a lower
average latency compared to recent latent code optimization methods, e.g., 4
seconds faster than D&B on the COCO-Subject dataset. Furthermore, through
visual comparisons and evaluation on the CLIP-IQA-Real metric, we show that
FRAP not only improves prompt-image alignment but also generates more authentic
images with realistic appearances. We also explore combining FRAP with prompt
rewriting LLM to recover their degraded prompt-image alignment, where we
observe improvements in both prompt-image alignment and image quality.
GRAB: A Challenging GRaph Analysis Benchmark for Large Multimodal Models
Authors: name: Jonathan Roberts, name: Kai Han, name: Samuel Albanie
Published: 2024-08-21 17:59:32

Large multimodal models (LMMs) have exhibited proficiencies across many
visual tasks. Although numerous well-known benchmarks exist to evaluate model
performance, they increasingly have insufficient headroom. As such, there is a
pressing need for a new generation of benchmarks challenging enough for the
next generation of LMMs. One area that LMMs show potential is graph analysis,
specifically, the tasks an analyst might typically perform when interpreting
figures such as estimating the mean, intercepts or correlations of functions
and data series. In this work, we introduce GRAB, a graph analysis benchmark,
fit for current and future frontier LMMs. Our benchmark is entirely synthetic,
ensuring high-quality, noise-free questions. GRAB is comprised of 2170
questions, covering four tasks and 23 graph properties. We evaluate 20 LMMs on
GRAB, finding it to be a challenging benchmark, with the highest performing
model attaining a score of just 21.7%. Finally, we conduct various ablations to
investigate where the models succeed and struggle. We release GRAB to encourage
progress in this important, growing domain.
Expanding FLORES+ Benchmark for more Low-Resource Settings:
  Portuguese-Emakhuwa Machine Translation Evaluation
Authors: name: Felermino D. M. Antonio Ali, name: Henrique Lopes Cardoso, name: Rui Sousa-Silva
Published: 2024-08-21 09:23:20

As part of the Open Language Data Initiative shared tasks, we have expanded
the FLORES+ evaluation set to include Emakhuwa, a low-resource language widely
spoken in Mozambique. We translated the dev and devtest sets from Portuguese
into Emakhuwa, and we detail the translation process and quality assurance
measures used. Our methodology involved various quality checks, including
post-editing and adequacy assessments. The resulting datasets consist of
multiple reference sentences for each source. We present baseline results from
training a Neural Machine Translation system and fine-tuning existing
multilingual translation models. Our findings suggest that spelling
inconsistencies remain a challenge in Emakhuwa. Additionally, the baseline
models underperformed on this evaluation set, underscoring the necessity for
further research to enhance machine translation quality for Emakhuwa. The data
is publicly available at https://huggingface.co/datasets/LIACC/Emakhuwa-FLORES.
LLM Pruning and Distillation in Practice: The Minitron Approach
Authors: name: Sharath Turuvekere Sreenivas, name: Saurav Muralidharan, name: Raviraj Joshi, name: Marcin Chochowski, name: Mostofa Patwary, name: Mohammad Shoeybi, name: Bryan Catanzaro, name: Jan Kautz, name: Pavlo Molchanov
Published: 2024-08-21 17:38:48

We present a comprehensive report on compressing the Llama 3.1 8B and Mistral
NeMo 12B models to 4B and 8B parameters, respectively, using pruning and
distillation. We explore two distinct pruning strategies: (1) depth pruning and
(2) joint hidden/attention/MLP (width) pruning, and evaluate the results on
common benchmarks from the LM Evaluation Harness. The models are then aligned
with NeMo Aligner and tested in instruct-tuned versions. This approach produces
a compelling 4B model from Llama 3.1 8B and a state-of-the-art
Mistral-NeMo-Minitron-8B (MN-Minitron-8B for brevity) model from Mistral NeMo
12B. We found that with no access to the original data, it is beneficial to
slightly fine-tune teacher models on the distillation dataset. We open-source
our base model weights on Hugging Face with a permissive license.
FocusLLM: Scaling LLM's Context by Parallel Decoding
Authors: name: Zhenyu Li, name: Yike Zhang, name: Tengyu Pan, name: Yutao Sun, name: Zhichao Duan, name: Junjie Fang, name: Rong Han, name: Zixuan Wang, name: Jianyong Wang
Published: 2024-08-21 16:11:59

Empowering LLMs with the ability to utilize useful information from a long
context is crucial for many downstream applications. However, achieving long
context lengths with the conventional transformer architecture requires
substantial training and inference resources. In this paper, we present
FocusLLM, a framework designed to extend the context length of any decoder-only
LLM, enabling the model to focus on relevant information from very long
sequences. FocusLLM processes long text inputs by dividing them into chunks
based on the model's original context length to alleviate the issue of
attention distraction. Then, it appends the local context to each chunk as a
prompt to extract essential information from each chunk based on a novel
parallel decoding mechanism, and ultimately integrates the extracted
information into the local context. FocusLLM stands out for great training
efficiency and versatility: trained with an 8K input length with much less
training cost than previous methods, FocusLLM exhibits superior performance
across downstream long-context tasks and maintains strong language modeling
ability when handling extensive long texts, even up to 400K tokens. Our code is
available at https://github.com/leezythu/FocusLLM.
TWLV-I: Analysis and Insights from Holistic Evaluation on Video
  Foundation Models
Authors: name: Hyeongmin Lee, name: Jin-Young Kim, name: Kyungjune Baek, name: Jihwan Kim, name: Hyojun Go, name: Seongsu Ha, name: Seokjin Han, name: Jiho Jang, name: Raehyuk Jung, name: Daewoo Kim, name: GeunOh Kim, name: JongMok Kim, name: Jongseok Kim, name: Junwan Kim, name: Soonwoo Kwon, name: Jangwon Lee, name: Seungjoon Park, name: Minjoon Seo, name: Jay Suh, name: Jaehyuk Yi, name: Aiden Lee
Published: 2024-08-21 03:56:27

In this work, we discuss evaluating video foundation models in a fair and
robust manner. Unlike language or image foundation models, many video
foundation models are evaluated with differing parameters (such as sampling
rate, number of frames, pretraining steps, etc.), making fair and robust
comparisons challenging. Therefore, we present a carefully designed evaluation
framework for measuring two core capabilities of video comprehension:
appearance and motion understanding. Our findings reveal that existing video
foundation models, whether text-supervised like UMT or InternVideo2, or
self-supervised like V-JEPA, exhibit limitations in at least one of these
capabilities. As an alternative, we introduce TWLV-I, a new video foundation
model that constructs robust visual representations for both motion- and
appearance-based videos. Based on the average top-1 accuracy of linear probing
on five action recognition benchmarks, pretrained only on publicly accessible
datasets, our model shows a 4.6%p improvement compared to V-JEPA (ViT-L) and a
7.7%p improvement compared to UMT (ViT-L). Even when compared to much larger
models, our model demonstrates a 7.2%p improvement compared to DFN (ViT-H), a
2.7%p improvement compared to V-JEPA~(ViT-H) and a 2.8%p improvement compared
to InternVideo2 (ViT-g). We provide embedding vectors obtained by TWLV-I from
videos of several commonly used video benchmarks, along with evaluation source
code that can directly utilize these embeddings. The code is available on
"https://github.com/twelvelabs-io/video-embeddings-evaluation-framework".