Orionnnn/hub-stats huggingface dataset:
GenCA: A Text-conditioned Generative Model for Realistic and Drivable
  Codec Avatars
Authors: name: Keqiang Sun, name: Amin Jourabloo, name: Riddhish Bhalodia, name: Moustafa Meshry, name: Yu Rong, name: Zhengyu Yang, name: Thu Nguyen-Phuoc, name: Christian Haene, name: Jiu Xu, name: Sam Johnson, name: Hongsheng Li, name: Sofien Bouaziz
Published: 2024-08-24 21:25:22

Photo-realistic and controllable 3D avatars are crucial for various
applications such as virtual and mixed reality (VR/MR), telepresence, gaming,
and film production. Traditional methods for avatar creation often involve
time-consuming scanning and reconstruction processes for each avatar, which
limits their scalability. Furthermore, these methods do not offer the
flexibility to sample new identities or modify existing ones. On the other
hand, by learning a strong prior from data, generative models provide a
promising alternative to traditional reconstruction methods, easing the time
constraints for both data capture and processing. Additionally, generative
methods enable downstream applications beyond reconstruction, such as editing
and stylization. Nonetheless, the research on generative 3D avatars is still in
its infancy, and therefore current methods still have limitations such as
creating static avatars, lacking photo-realism, having incomplete facial
details, or having limited drivability. To address this, we propose a
text-conditioned generative model that can generate photo-realistic facial
avatars of diverse identities, with more complete details like hair, eyes and
mouth interior, and which can be driven through a powerful non-parametric
latent expression space. Specifically, we integrate the generative and editing
capabilities of latent diffusion models with a strong prior model for avatar
expression driving.
  Our model can generate and control high-fidelity avatars, even those
out-of-distribution. We also highlight its potential for downstream
applications, including avatar editing and single-shot avatar reconstruction.
Platypus: A Generalized Specialist Model for Reading Text in Various
  Forms
Authors: name: Peng Wang, name: Zhaohai Li, name: Jun Tang, name: Humen Zhong, name: Fei Huang, name: Zhibo Yang, name: Cong Yao
Published: 2024-08-27 06:24:51

Reading text from images (either natural scenes or documents) has been a
long-standing research topic for decades, due to the high technical challenge
and wide application range. Previously, individual specialist models are
developed to tackle the sub-tasks of text reading (e.g., scene text
recognition, handwritten text recognition and mathematical expression
recognition). However, such specialist models usually cannot effectively
generalize across different sub-tasks. Recently, generalist models (such as
GPT-4V), trained on tremendous data in a unified way, have shown enormous
potential in reading text in various scenarios, but with the drawbacks of
limited accuracy and low efficiency. In this work, we propose Platypus, a
generalized specialist model for text reading. Specifically, Platypus combines
the best of both worlds: being able to recognize text of various forms with a
single unified architecture, while achieving excellent accuracy and high
efficiency. To better exploit the advantage of Platypus, we also construct a
text reading dataset (called Worms), the images of which are curated from
previous datasets and partially re-labeled. Experiments on standard benchmarks
demonstrate the effectiveness and superiority of the proposed Platypus model.
Model and data will be made publicly available at
https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/Platypus.
NanoFlow: Towards Optimal Large Language Model Serving Throughput
Authors: name: Kan Zhu, name: Yilong Zhao, name: Liangyu Zhao, name: Gefei Zuo, name: Yile Gu, name: Dedong Xie, name: Yufei Gao, name: Qinyu Xu, name: Tian Tang, name: Zihao Ye, name: Keisuke Kamahori, name: Chien-Yu Lin, name: Stephanie Wang, name: Arvind Krishnamurthy, name: Baris Kasikci
Published: 2024-08-22 23:00:40

The increasing usage of Large Language Models (LLMs) has resulted in a
surging demand for planet-scale serving systems, where tens of thousands of
GPUs continuously serve hundreds of millions of users. Consequently, throughput
(under reasonable latency constraints) has emerged as a key metric that
determines serving systems' performance. To boost throughput, various methods
of inter-device parallelism (e.g., data, tensor, pipeline) have been explored.
However, existing methods do not consider overlapping the utilization of
different resources within a single device, leading to underutilization and
sub-optimal performance.
  We propose NanoFlow, a novel serving framework that exploits intra-device
parallelism, which overlaps the usage of resources including compute, memory,
and network within a single device through operation co-scheduling. To exploit
intra-device parallelism, NanoFlow introduces two key innovations: First,
NanoFlow splits requests into nano-batches at the granularity of operations,
which breaks the dependency of sequential operations in LLM inference and
enables overlapping; then, to get benefit from overlapping, NanoFlow uses an
operation-level pipeline with execution unit scheduling, which partitions the
device's functional units and simultaneously executes different operations in
each unit. NanoFlow automates the pipeline setup using a parameter search
algorithm, which enables easily porting NanoFlow to different models. We
implement NanoFlow on NVIDIA GPUs and evaluate end-to-end serving throughput on
several popular models such as LLaMA-2-70B, Mixtral 8x7B, LLaMA-3-8B, etc..
With practical workloads, NanoFlow provides 1.91x throughput boost compared to
state-of-the-art serving systems achieving 59% to 72% of optimal throughput
across ported models.
MagicMan: Generative Novel View Synthesis of Humans with 3D-Aware
  Diffusion and Iterative Refinement
Authors: name: Xu He, name: Xiaoyu Li, name: Di Kang, name: Jiangnan Ye, name: Chaopeng Zhang, name: Liyang Chen, name: Xiangjun Gao, name: Han Zhang, name: Zhiyong Wu, name: Haolin Zhuang
Published: 2024-08-26 12:10:52

Existing works in single-image human reconstruction suffer from weak
generalizability due to insufficient training data or 3D inconsistencies for a
lack of comprehensive multi-view knowledge. In this paper, we introduce
MagicMan, a human-specific multi-view diffusion model designed to generate
high-quality novel view images from a single reference image. As its core, we
leverage a pre-trained 2D diffusion model as the generative prior for
generalizability, with the parametric SMPL-X model as the 3D body prior to
promote 3D awareness. To tackle the critical challenge of maintaining
consistency while achieving dense multi-view generation for improved 3D human
reconstruction, we first introduce hybrid multi-view attention to facilitate
both efficient and thorough information interchange across different views.
Additionally, we present a geometry-aware dual branch to perform concurrent
generation in both RGB and normal domains, further enhancing consistency via
geometry cues. Last but not least, to address ill-shaped issues arising from
inaccurate SMPL-X estimation that conflicts with the reference image, we
propose a novel iterative refinement strategy, which progressively optimizes
SMPL-X accuracy while enhancing the quality and consistency of the generated
multi-views. Extensive experimental results demonstrate that our method
significantly outperforms existing approaches in both novel view synthesis and
subsequent 3D human reconstruction tasks.
Foundation Models for Music: A Survey
Authors: name: Yinghao Ma, name: Anders Øland, name: Anton Ragni, name: Bleiz MacSen Del Sette, name: Charalampos Saitis, name: Chris Donahue, name: Chenghua Lin, name: Christos Plachouras, name: Emmanouil Benetos, name: Elio Quinton, name: Elona Shatri, name: Fabio Morreale, name: Ge Zhang, name: György Fazekas, name: Gus Xia, name: Huan Zhang, name: Ilaria Manco, name: Jiawen Huang, name: Julien Guinot, name: Liwei Lin, name: Luca Marinelli, name: Max W. Y. Lam, name: Megha Sharma, name: Qiuqiang Kong, name: Roger B. Dannenberg, name: Ruibin Yuan, name: Shangda Wu, name: Shih-Lun Wu, name: Shuqi Dai, name: Shun Lei, name: Shiyin Kang, name: Simon Dixon, name: Wenhu Chen, name: Wehhao Huang, name: Xingjian Du, name: Xingwei Qu, name: Xu Tan, name: Yizhi Li, name: Zeyue Tian, name: Zhiyong Wu, name: Zhizheng Wu, name: Ziyang Ma, name: Ziyu Wang
Published: 2024-08-26 15:13:14

In recent years, foundation models (FMs) such as large language models (LLMs)
and latent diffusion models (LDMs) have profoundly impacted diverse sectors,
including music. This comprehensive review examines state-of-the-art (SOTA)
pre-trained models and foundation models in music, spanning from representation
learning, generative learning and multimodal learning. We first contextualise
the significance of music in various industries and trace the evolution of AI
in music. By delineating the modalities targeted by foundation models, we
discover many of the music representations are underexplored in FM development.
Then, emphasis is placed on the lack of versatility of previous methods on
diverse music applications, along with the potential of FMs in music
understanding, generation and medical application. By comprehensively exploring
the details of the model pre-training paradigm, architectural choices,
tokenisation, finetuning methodologies and controllability, we emphasise the
important topics that should have been well explored, like instruction tuning
and in-context learning, scaling law and emergent ability, as well as
long-sequence modelling etc. A dedicated section presents insights into music
agents, accompanied by a thorough analysis of datasets and evaluations
essential for pre-training and downstream tasks. Finally, by underscoring the
vital importance of ethical considerations, we advocate that following research
on FM for music should focus more on such issues as interpretability,
transparency, human responsibility, and copyright issues. The paper offers
insights into future challenges and trends on FMs for music, aiming to shape
the trajectory of human-AI collaboration in the music realm.
SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
  Teacher
Authors: name: Trung Dao, name: Thuan Hoang Nguyen, name: Thanh Le, name: Duc Vu, name: Khoi Nguyen, name: Cuong Pham, name: Anh Tran
Published: 2024-08-26 10:42:53

In this paper, we aim to enhance the performance of SwiftBrush, a prominent
one-step text-to-image diffusion model, to be competitive with its multi-step
Stable Diffusion counterpart. Initially, we explore the quality-diversity
trade-off between SwiftBrush and SD Turbo: the former excels in image
diversity, while the latter excels in image quality. This observation motivates
our proposed modifications in the training methodology, including better weight
initialization and efficient LoRA training. Moreover, our introduction of a
novel clamped CLIP loss enhances image-text alignment and results in improved
image quality. Remarkably, by combining the weights of models trained with
efficient LoRA and full training, we achieve a new state-of-the-art one-step
diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
multi-step Stable Diffusion models. The evaluation code is available at:
https://github.com/vinairesearch/swiftbrushv2.
Learning to Move Like Professional Counter-Strike Players
Authors: name: David Durst, name: Feng Xie, name: Vishnu Sarukkai, name: Brennan Shacklett, name: Iuri Frosio, name: Chen Tessler, name: Joohwan Kim, name: Carly Taylor, name: Gilbert Bernstein, name: Sanjiban Choudhury, name: Pat Hanrahan, name: Kayvon Fatahalian
Published: 2024-08-25 20:43:34

In multiplayer, first-person shooter games like Counter-Strike: Global
Offensive (CS:GO), coordinated movement is a critical component of high-level
strategic play. However, the complexity of team coordination and the variety of
conditions present in popular game maps make it impractical to author
hand-crafted movement policies for every scenario. We show that it is possible
to take a data-driven approach to creating human-like movement controllers for
CS:GO. We curate a team movement dataset comprising 123 hours of professional
game play traces, and use this dataset to train a transformer-based movement
model that generates human-like team movement for all players in a "Retakes"
round of the game. Importantly, the movement prediction model is efficient.
Performing inference for all players takes less than 0.5 ms per game step
(amortized cost) on a single CPU core, making it plausible for use in
commercial games today. Human evaluators assess that our model behaves more
like humans than both commercially-available bots and procedural movement
controllers scripted by experts (16% to 59% higher by TrueSkill rating of
"human-like"). Using experiments involving in-game bot vs. bot self-play, we
demonstrate that our model performs simple forms of teamwork, makes fewer
common movement mistakes, and yields movement distributions, player lifetimes,
and kill locations similar to those observed in professional CS:GO match play.
MobileQuant: Mobile-friendly Quantization for On-device Language Models
Authors: name: Fuwen Tan, name: Royson Lee, name: Łukasz Dudziak, name: Shell Xu Hu, name: Sourav Bhattacharya, name: Timothy Hospedales, name: Georgios Tzimiropoulos, name: Brais Martinez
Published: 2024-08-25 20:41:22

Large language models (LLMs) have revolutionized language processing,
delivering outstanding results across multiple applications. However, deploying
LLMs on edge devices poses several challenges with respect to memory, energy,
and compute costs, limiting their widespread use in devices such as mobile
phones. A promising solution is to reduce the number of bits used to represent
weights and activations. While existing works have found partial success at
quantizing LLMs to lower bitwidths, e.g. 4-bit weights, quantizing activations
beyond 16 bits often leads to large computational overheads due to poor
on-device quantization support, or a considerable accuracy drop. Yet, 8-bit
activations are very attractive for on-device deployment as they would enable
LLMs to fully exploit mobile-friendly hardware, e.g. Neural Processing Units
(NPUs). In this work, we make a first attempt to facilitate the on-device
deployment of LLMs using integer-only quantization. We first investigate the
limitations of existing quantization methods for on-device deployment, with a
special focus on activation quantization. We then address these limitations by
introducing a simple post-training quantization method, named MobileQuant, that
extends previous weight equivalent transformation works by jointly optimizing
the weight transformation and activation range parameters in an end-to-end
manner. MobileQuant demonstrates superior capabilities over existing methods by
1) achieving near-lossless quantization on a wide range of LLM benchmarks, 2)
reducing latency and energy consumption by 20\%-50\% compared to current
on-device quantization strategies, 3) requiring limited compute budget, 4)
being compatible with mobile-friendly compute units, e.g. NPU.
K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models
  via K-wise Human Preferences
Authors: name: Zhikai Li, name: Xuewen Liu, name: Dongrong Fu, name: Jianquan Li, name: Qingyi Gu, name: Kurt Keutzer, name: Zhen Dong
Published: 2024-08-26 17:58:20

The rapid advancement of visual generative models necessitates efficient and
reliable evaluation methods. Arena platform, which gathers user votes on model
comparisons, can rank models with human preferences. However, traditional Arena
methods, while established, require an excessive number of comparisons for
ranking to converge and are vulnerable to preference noise in voting,
suggesting the need for better approaches tailored to contemporary evaluation
challenges. In this paper, we introduce K-Sort Arena, an efficient and reliable
platform based on a key insight: images and videos possess higher perceptual
intuitiveness than texts, enabling rapid evaluation of multiple samples
simultaneously. Consequently, K-Sort Arena employs K-wise comparisons, allowing
K models to engage in free-for-all competitions, which yield much richer
information than pairwise comparisons. To enhance the robustness of the system,
we leverage probabilistic modeling and Bayesian updating techniques. We propose
an exploration-exploitation-based matchmaking strategy to facilitate more
informative comparisons. In our experiments, K-Sort Arena exhibits 16.3x faster
convergence compared to the widely used ELO algorithm. To further validate the
superiority and obtain a comprehensive leaderboard, we collect human feedback
via crowdsourced evaluations of numerous cutting-edge text-to-image and
text-to-video models. Thanks to its high efficiency, K-Sort Arena can
continuously incorporate emerging models and update the leaderboard with
minimal votes. Our project has undergone several months of internal testing and
is now available at https://huggingface.co/spaces/ksort/K-Sort-Arena
LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to
  Small-Scale Local LLMs
Authors: name: Chansung Park, name: Juyong Jiang, name: Fan Wang, name: Sayak Paul, name: Jing Tang, name: Sunghun Kim
Published: 2024-08-24 05:03:08

The widespread adoption of cloud-based proprietary large language models
(LLMs) has introduced significant challenges, including operational
dependencies, privacy concerns, and the necessity of continuous internet
connectivity. In this work, we introduce an LLMOps pipeline, "LlamaDuo", for
the seamless migration of knowledge and abilities from service-oriented LLMs to
smaller, locally manageable models. This pipeline is crucial for ensuring
service continuity in the presence of operational failures, strict privacy
policies, or offline requirements. Our LlamaDuo involves fine-tuning a small
language model against the service LLM using a synthetic dataset generated by
the latter. If the performance of the fine-tuned model falls short of
expectations, it is enhanced by further fine-tuning with additional similar
data created by the service LLM. This iterative process guarantees that the
smaller model can eventually match or even surpass the service LLM's
capabilities in specific downstream tasks, offering a practical and scalable
solution for managing AI deployments in constrained environments. Extensive
experiments with leading edge LLMs are conducted to demonstrate the
effectiveness, adaptability, and affordability of LlamaDuo across various
downstream tasks. Our pipeline implementation is available at
https://github.com/deep-diver/llamaduo.
LLaVaOLMoBitnet1B: Ternary LLM goes Multimodal!
Authors: name: Jainaveen Sundaram, name: Ravishankar Iyer
Published: 2024-08-23 23:00:19

Multimodal Large Language Models (MM-LLMs) have seen significant advancements
in the last year, demonstrating impressive performance across tasks. However,
to truly democratize AI, models must exhibit strong capabilities and be able to
run efficiently on small compute footprints accessible by most. Part of this
quest, we introduce LLaVaOLMoBitnet1B - the first Ternary Multimodal LLM
capable of accepting Image(s)+Text inputs to produce coherent textual
responses. The model is fully open-sourced along with training scripts to
encourage further research in this space. This accompanying technical report
highlights the training process, evaluation details, challenges associated with
ternary models and future opportunities. Link to the model:
https://huggingface.co/IntelLabs/LlavaOLMoBitnet1B
Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate
  Scheduler
Authors: name: Yikang Shen, name: Matthew Stallone, name: Mayank Mishra, name: Gaoyuan Zhang, name: Shawn Tan, name: Aditya Prasad, name: Adriana Meza Soria, name: David D. Cox, name: Rameswar Panda
Published: 2024-08-23 20:22:20

Finding the optimal learning rate for language model pretraining is a
challenging task. This is not only because there is a complicated correlation
between learning rate, batch size, number of training tokens, model size, and
other hyperparameters but also because it is prohibitively expensive to perform
a hyperparameter search for large language models with Billions or Trillions of
parameters. Recent studies propose using small proxy models and small corpus to
perform hyperparameter searches and transposing the optimal parameters to large
models and large corpus. While the zero-shot transferability is theoretically
and empirically proven for model size related hyperparameters, like depth and
width, the zero-shot transfer from small corpus to large corpus is
underexplored. In this paper, we study the correlation between optimal learning
rate, batch size, and number of training tokens for the recently proposed WSD
scheduler. After thousands of small experiments, we found a power-law
relationship between variables and demonstrated its transferability across
model sizes. Based on the observation, we propose a new learning rate
scheduler, Power scheduler, that is agnostic about the number of training
tokens and batch size. The experiment shows that combining the Power scheduler
with Maximum Update Parameterization (muP) can consistently achieve impressive
performance with one set of hyperparameters regardless of the number of
training tokens, batch size, model size, and even model architecture. Our 3B
dense and MoE models trained with the Power scheduler achieve comparable
performance as state-of-the-art small language models. We open-source these
pretrained models at https://ibm.biz/BdKhLa.
SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
Authors: name: Daoguang Zan, name: Zhirong Huang, name: Ailun Yu, name: Shaoxin Lin, name: Yifan Shi, name: Wei Liu, name: Dong Chen, name: Zongshuai Qi, name: Hao Yu, name: Lei Yu, name: Dezhi Ran, name: Muhan Zeng, name: Bo Shen, name: Pan Bian, name: Guangtai Liang, name: Bei Guan, name: Pengjie Huang, name: Tao Xie, name: Yongji Wang, name: Qianxiang Wang
Published: 2024-08-26 15:30:05

GitHub issue resolving is a critical task in software engineering, recently
gaining significant attention in both industry and academia. Within this task,
SWE-bench has been released to evaluate issue resolving capabilities of large
language models (LLMs), but has so far only focused on Python version. However,
supporting more programming languages is also important, as there is a strong
demand in industry. As a first step toward multilingual support, we have
developed a Java version of SWE-bench, called SWE-bench-java. We have publicly
released the dataset, along with the corresponding Docker-based evaluation
environment and leaderboard, which will be continuously maintained and updated
in the coming months. To verify the reliability of SWE-bench-java, we implement
a classic method SWE-agent and test several powerful LLMs on it. As is well
known, developing a high-quality multi-lingual benchmark is time-consuming and
labor-intensive, so we welcome contributions through pull requests or
collaboration to accelerate its iteration and refinement, paving the way for
fully automated programming.