Orionnnn/hub-stats huggingface dataset:
Nemotron-4 15B Technical Report
Authors: name: Jupinder Parmar, name: Shrimai Prabhumoye, name: Joseph Jennings, name: Mostofa Patwary, name: Sandeep Subramanian, name: Dan Su, name: Chen Zhu, name: Deepak Narayanan, name: Aastha Jhunjhunwala, name: Ayush Dattagupta, name: Vibhu Jawa, name: Jiwei Liu, name: Ameya Mahabaleshwarkar, name: Osvald Nitski, name: Annika Brundyn, name: James Maki, name: Miguel Martinez, name: Jiaxuan You, name: John Kamalu, name: Patrick LeGresley, name: Denys Fridman, name: Jared Casper, name: Ashwath Aithal, name: Oleksii Kuchaiev, name: Mohammad Shoeybi, name: Jonathan Cohen, name: Bryan Catanzaro
Published: 2024-02-26 18:43:45

We introduce Nemotron-4 15B, a 15-billion-parameter large multilingual
language model trained on 8 trillion text tokens. Nemotron-4 15B demonstrates
strong performance when assessed on English, multilingual, and coding tasks: it
outperforms all existing similarly-sized open models on 4 out of 7 downstream
evaluation areas and achieves competitive performance to the leading open
models in the remaining ones. Specifically, Nemotron-4 15B exhibits the best
multilingual capabilities of all similarly-sized models, even outperforming
models over four times larger and those explicitly specialized for multilingual
tasks.
Towards Open-ended Visual Quality Comparison
Authors: name: Haoning Wu, name: Hanwei Zhu, name: Zicheng Zhang, name: Erli Zhang, name: Chaofeng Chen, name: Liang Liao, name: Chunyi Li, name: Annan Wang, name: Wenxiu Sun, name: Qiong Yan, name: Xiaohong Liu, name: Guangtao Zhai, name: Shiqi Wang, name: Weisi Lin
Published: 2024-02-26 15:10:56

Comparative settings (e.g. pairwise choice, listwise ranking) have been
adopted by a wide range of subjective studies for image quality assessment
(IQA), as it inherently standardizes the evaluation criteria across different
observers and offer more clear-cut responses. In this work, we extend the edge
of emerging large multi-modality models (LMMs) to further advance visual
quality comparison into open-ended settings, that 1) can respond to open-range
questions on quality comparison; 2) can provide detailed reasonings beyond
direct answers. To this end, we propose the Co-Instruct. To train this
first-of-its-kind open-source open-ended visual quality comparer, we collect
the Co-Instruct-562K dataset, from two sources: (a) LMM-merged single image
quality description, (b) GPT-4V "teacher" responses on unlabeled data.
Furthermore, to better evaluate this setting, we propose the MICBench, the
first benchmark on multi-image comparison for LMMs. We demonstrate that
Co-Instruct not only achieves 30% higher superior accuracy than
state-of-the-art open-source LMMs, but also outperforms GPT-4V (its teacher),
on both existing related benchmarks and the proposed MICBench. Our model is
published at https://huggingface.co/q-future/co-instruct.
ChatMusician: Understanding and Generating Music Intrinsically with LLM
Authors: name: Ruibin Yuan, name: Hanfeng Lin, name: Yi Wang, name: Zeyue Tian, name: Shangda Wu, name: Tianhao Shen, name: Ge Zhang, name: Yuhang Wu, name: Cong Liu, name: Ziya Zhou, name: Ziyang Ma, name: Liumeng Xue, name: Ziyu Wang, name: Qin Liu, name: Tianyu Zheng, name: Yizhi Li, name: Yinghao Ma, name: Yiming Liang, name: Xiaowei Chi, name: Ruibo Liu, name: Zili Wang, name: Pengfei Li, name: Jingcheng Wu, name: Chenghua Lin, name: Qifeng Liu, name: Tao Jiang, name: Wenhao Huang, name: Wenhu Chen, name: Emmanouil Benetos, name: Jie Fu, name: Gus Xia, name: Roger Dannenberg, name: Wei Xue, name: Shiyin Kang, name: Yike Guo
Published: 2024-02-25 17:19:41

While Large Language Models (LLMs) demonstrate impressive capabilities in
text generation, we find that their ability has yet to be generalized to music,
humanity's creative language. We introduce ChatMusician, an open-source LLM
that integrates intrinsic musical abilities. It is based on continual
pre-training and finetuning LLaMA2 on a text-compatible music representation,
ABC notation, and the music is treated as a second language. ChatMusician can
understand and generate music with a pure text tokenizer without any external
multi-modal neural structures or tokenizers. Interestingly, endowing musical
abilities does not harm language abilities, even achieving a slightly higher
MMLU score. Our model is capable of composing well-structured, full-length
music, conditioned on texts, chords, melodies, motifs, musical forms, etc,
surpassing GPT-4 baseline. On our meticulously curated college-level music
understanding benchmark, MusicTheoryBench, ChatMusician surpasses LLaMA2 and
GPT-3.5 on zero-shot setting by a noticeable margin. Our work reveals that LLMs
can be an excellent compressor for music, but there remains significant
territory to be conquered. We release our 4B token music-language corpora
MusicPile, the collected MusicTheoryBench, code, model and demo in GitHub.
Multi-LoRA Composition for Image Generation
Authors: name: Ming Zhong, name: Yelong Shen, name: Shuohang Wang, name: Yadong Lu, name: Yizhu Jiao, name: Siru Ouyang, name: Donghan Yu, name: Jiawei Han, name: Weizhu Chen
Published: 2024-02-26 18:59:18

Low-Rank Adaptation (LoRA) is extensively utilized in text-to-image models
for the accurate rendition of specific elements like distinct characters or
unique styles in generated images. Nonetheless, existing methods face
challenges in effectively composing multiple LoRAs, especially as the number of
LoRAs to be integrated grows, thus hindering the creation of complex imagery.
In this paper, we study multi-LoRA composition through a decoding-centric
perspective. We present two training-free methods: LoRA Switch, which
alternates between different LoRAs at each denoising step, and LoRA Composite,
which simultaneously incorporates all LoRAs to guide more cohesive image
synthesis. To evaluate the proposed approaches, we establish ComposLoRA, a new
comprehensive testbed as part of this research. It features a diverse range of
LoRA categories with 480 composition sets. Utilizing an evaluation framework
based on GPT-4V, our findings demonstrate a clear improvement in performance
with our methods over the prevalent baseline, particularly evident when
increasing the number of LoRAs in a composition.
Seamless Human Motion Composition with Blended Positional Encodings
Authors: name: German Barquero, name: Sergio Escalera, name: Cristina Palmero
Published: 2024-02-23 18:59:40

Conditional human motion generation is an important topic with many
applications in virtual reality, gaming, and robotics. While prior works have
focused on generating motion guided by text, music, or scenes, these typically
result in isolated motions confined to short durations. Instead, we address the
generation of long, continuous sequences guided by a series of varying textual
descriptions. In this context, we introduce FlowMDM, the first diffusion-based
model that generates seamless Human Motion Compositions (HMC) without any
postprocessing or redundant denoising steps. For this, we introduce the Blended
Positional Encodings, a technique that leverages both absolute and relative
positional encodings in the denoising chain. More specifically, global motion
coherence is recovered at the absolute stage, whereas smooth and realistic
transitions are built at the relative stage. As a result, we achieve
state-of-the-art results in terms of accuracy, realism, and smoothness on the
Babel and HumanML3D datasets. FlowMDM excels when trained with only a single
description per motion sequence thanks to its Pose-Centric Cross-ATtention,
which makes it robust against varying text descriptions at inference time.
Finally, to address the limitations of existing HMC metrics, we propose two new
metrics: the Peak Jerk and the Area Under the Jerk, to detect abrupt
transitions.
Genie: Generative Interactive Environments
Authors: name: Jake Bruce, name: Michael Dennis, name: Ashley Edwards, name: Jack Parker-Holder, name: Yuge Shi, name: Edward Hughes, name: Matthew Lai, name: Aditi Mavalankar, name: Richie Steigerwald, name: Chris Apps, name: Yusuf Aytar, name: Sarah Bechtle, name: Feryal Behbahani, name: Stephanie Chan, name: Nicolas Heess, name: Lucy Gonzalez, name: Simon Osindero, name: Sherjil Ozair, name: Scott Reed, name: Jingwei Zhang, name: Konrad Zolna, name: Jeff Clune, name: Nando de Freitas, name: Satinder Singh, name: Tim Rocktäschel
Published: 2024-02-23 15:47:26

We introduce Genie, the first generative interactive environment trained in
an unsupervised manner from unlabelled Internet videos. The model can be
prompted to generate an endless variety of action-controllable virtual worlds
described through text, synthetic images, photographs, and even sketches. At
11B parameters, Genie can be considered a foundation world model. It is
comprised of a spatiotemporal video tokenizer, an autoregressive dynamics
model, and a simple and scalable latent action model. Genie enables users to
act in the generated environments on a frame-by-frame basis despite training
without any ground-truth action labels or other domain-specific requirements
typically found in the world model literature. Further the resulting learned
latent action space facilitates training agents to imitate behaviors from
unseen videos, opening the path for training generalist agents of the future.
AgentOhana: Design Unified Data and Training Pipeline for Effective
  Agent Learning
Authors: name: Jianguo Zhang, name: Tian Lan, name: Rithesh Murthy, name: Zhiwei Liu, name: Weiran Yao, name: Juntao Tan, name: Thai Hoang, name: Liangwei Yang, name: Yihao Feng, name: Zuxin Liu, name: Tulika Awalgaonkar, name: Juan Carlos Niebles, name: Silvio Savarese, name: Shelby Heinecke, name: Huan Wang, name: Caiming Xiong
Published: 2024-02-23 18:56:26

Autonomous agents powered by large language models (LLMs) have garnered
significant research attention. However, fully harnessing the potential of LLMs
for agent-based tasks presents inherent challenges due to the heterogeneous
nature of diverse data sources featuring multi-turn trajectories. In this
paper, we introduce AgentOhana as a comprehensive solution to address
these challenges. AgentOhana aggregates agent trajectories from
distinct environments, spanning a wide array of scenarios. It meticulously
standardizes and unifies these trajectories into a consistent format,
streamlining the creation of a generic data loader optimized for agent
training. Leveraging the data unification, our training pipeline maintains
equilibrium across different data sources and preserves independent randomness
across devices during dataset partitioning and model training. Additionally, we
present xLAM-v0.1, a large action model tailored for AI agents, which
demonstrates exceptional performance across various benchmarks.
API-BLEND: A Comprehensive Corpora for Training and Benchmarking API
  LLMs
Authors: name: Kinjal Basu, name: Ibrahim Abdelaziz, name: Subhajit Chaudhury, name: Soham Dan, name: Maxwell Crouse, name: Asim Munawar, name: Sadhana Kumaravel, name: Vinod Muthusamy, name: Pavan Kapanipathi, name: Luis A. Lastras
Published: 2024-02-23 18:30:49

There is a growing need for Large Language Models (LLMs) to effectively use
tools and external Application Programming Interfaces (APIs) to plan and
complete tasks. As such, there is tremendous interest in methods that can
acquire sufficient quantities of train and test data that involve calls to
tools / APIs. Two lines of research have emerged as the predominant strategies
for addressing this challenge. The first has focused on synthetic data
generation techniques, while the second has involved curating task-adjacent
datasets which can be transformed into API / Tool-based tasks. In this paper,
we focus on the task of identifying, curating, and transforming existing
datasets and, in turn, introduce API-BLEND, a large corpora for training and
systematic testing of tool-augmented LLMs. The datasets mimic real-world
scenarios involving API-tasks such as API / tool detection, slot filling, and
sequencing of the detected APIs. We demonstrate the utility of the API-BLEND
dataset for both training and benchmarking purposes.
GPTVQ: The Blessing of Dimensionality for LLM Quantization
Authors: name: Mart van Baalen, name: Andrey Kuzmin, name: Markus Nagel, name: Peter Couperus, name: Cedric Bastoul, name: Eric Mahurin, name: Tijmen Blankevoort, name: Paul Whatmough
Published: 2024-02-23 13:39:16

In this work we show that the size versus accuracy trade-off of neural
network quantization can be significantly improved by increasing the
quantization dimensionality. We propose the GPTVQ method, a new fast method for
post-training vector quantization (VQ) that scales well to Large Language
Models (LLMs). Our method interleaves quantization of one or more columns with
updates to the remaining unquantized weights, using information from the
Hessian of the per-layer output reconstruction MSE. Quantization codebooks are
initialized using an efficient data-aware version of the EM algorithm. The
codebooks are then updated, and further compressed by using integer
quantization and SVD-based compression. GPTVQ establishes a new state-of-the
art in the size vs accuracy trade-offs on a wide range of LLMs such as Llama-v2
and Mistral. Furthermore, our method is efficient: on a single H100 it takes
between 3 and 11 hours to process a Llamav2-70B model, depending on
quantization setting. Lastly, with on-device timings for VQ decompression on a
mobile CPU we show that VQ leads to improved latency compared to using a 4-bit
integer format.
ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and
  Two-Phase Partition
Authors: name: Lu Ye, name: Ze Tao, name: Yong Huang, name: Yang Li
Published: 2024-02-23 09:29:19

Self-attention is an essential component of large language models(LLMs) but a
significant source of inference latency for long sequences. In multi-tenant
LLMs serving scenarios, the compute and memory operation cost of self-attention
can be optimized by using the probability that multiple LLM requests have
shared system prompts in prefixes. In this paper, we introduce ChunkAttention,
a prefix-aware self-attention module that can detect matching prompt prefixes
across multiple requests and share their key/value tensors in memory at runtime
to improve the memory utilization of KV cache. This is achieved by breaking
monolithic key/value tensors into smaller chunks and structuring them into the
auxiliary prefix tree. Consequently, on top of the prefix-tree based KV cache,
we design an efficient self-attention kernel, where a two-phase partition
algorithm is implemented to improve the data locality during self-attention
computation in the presence of shared system prompts. Experiments show that
ChunkAttention can speed up the self-attention kernel by 3.2-4.8times
compared to the start-of-the-art implementation, with the length of the system
prompt ranging from 1024 to 4096.
Divide-or-Conquer? Which Part Should You Distill Your LLM?
Authors: name: Zhuofeng Wu, name: He Bai, name: Aonan Zhang, name: Jiatao Gu, name: VG Vinod Vydiswaran, name: Navdeep Jaitly, name: Yizhe Zhang
Published: 2024-02-22 22:28:46

Recent methods have demonstrated that Large Language Models (LLMs) can solve
reasoning tasks better when they are encouraged to solve subtasks of the main
task first. In this paper we devise a similar strategy that breaks down
reasoning tasks into a problem decomposition phase and a problem solving phase
and show that the strategy is able to outperform a single stage solution.
Further, we hypothesize that the decomposition should be easier to distill into
a smaller model compared to the problem solving because the latter requires
large amounts of domain knowledge while the former only requires learning
general problem solving strategies. We propose methods to distill these two
capabilities and evaluate their impact on reasoning outcomes and inference
cost. We find that we can distill the problem decomposition phase and at the
same time achieve good generalization across tasks, datasets, and models.
However, it is harder to distill the problem solving capability without losing
performance and the resulting distilled model struggles with generalization.
These results indicate that by using smaller, distilled problem decomposition
models in combination with problem solving LLMs we can achieve reasoning with
cost-efficient inference and local adaptation.
MobileLLM: Optimizing Sub-billion Parameter Language Models for
  On-Device Use Cases
Authors: name: Zechun Liu, name: Changsheng Zhao, name: Forrest Iandola, name: Chen Lai, name: Yuandong Tian, name: Igor Fedorov, name: Yunyang Xiong, name: Ernie Chang, name: Yangyang Shi, name: Raghuraman Krishnamoorthi, name: Liangzhen Lai, name: Vikas Chandra
Published: 2024-02-22 18:58:55

This paper addresses the growing need for efficient large language models
(LLMs) on mobile devices, driven by increasing cloud costs and latency
concerns. We focus on designing top-quality LLMs with fewer than a billion
parameters, a practical choice for mobile deployment. Contrary to prevailing
belief emphasizing the pivotal role of data and parameter quantity in
determining model quality, our investigation underscores the significance of
model architecture for sub-billion scale LLMs. Leveraging deep and thin
architectures, coupled with embedding sharing and grouped-query attention
mechanisms, we establish a strong baseline network denoted as MobileLLM, which
attains a remarkable 2.7%/4.3% accuracy boost over preceding 125M/350M
state-of-the-art models. Additionally, we propose an immediate block-wise
weight sharing approach with no increase in model size and only marginal
latency overhead. The resultant models, denoted as MobileLLM-LS, demonstrate a
further accuracy enhancement of 0.7%/0.8% than MobileLLM 125M/350M. Moreover,
MobileLLM model family shows significant improvements compared to previous
sub-billion models on chat benchmarks, and demonstrates close correctness to
LLaMA-v2 7B in API calling tasks, highlighting the capability of small models
for common on-device use cases.
Watermarking Makes Language Models Radioactive
Authors: name: Tom Sander, name: Pierre Fernandez, name: Alain Durmus, name: Matthijs Douze, name: Teddy Furon
Published: 2024-02-22 18:55:22

This paper investigates the radioactivity of LLM-generated texts, i.e.
whether it is possible to detect that such input was used as training data.
Conventional methods like membership inference can carry out this detection
with some level of accuracy. We show that watermarked training data leaves
traces easier to detect and much more reliable than membership inference. We
link the contamination level to the watermark robustness, its proportion in the
training set, and the fine-tuning process. We notably demonstrate that training
on watermarked synthetic instructions can be detected with high confidence
(p-value < 1e-5) even when as little as 5% of training text is watermarked.
Thus, LLM watermarking, originally designed for detecting machine-generated
text, gives the ability to easily identify if the outputs of a watermarked LLM
were used to fine-tune another LLM.
Same Task, More Tokens: the Impact of Input Length on the Reasoning
  Performance of Large Language Models
Authors: name: Mosh Levy, name: Alon Jacoby, name: Yoav Goldberg
Published: 2024-02-19 16:04:53

This paper explores the impact of extending input lengths on the capabilities
of Large Language Models (LLMs). Despite LLMs advancements in recent times,
their performance consistency across different input lengths is not well
understood. We investigate this aspect by introducing a novel QA reasoning
framework, specifically designed to assess the impact of input length. We
isolate the effect of input length using multiple versions of the same sample,
each being extended with padding of different lengths, types and locations. Our
findings show a notable degradation in LLMs' reasoning performance at much
shorter input lengths than their technical maximum. We show that the
degradation trend appears in every version of our dataset, although at
different intensities. Additionally, our study reveals that traditional
perplexity metrics do not correlate with performance of LLMs' in long input
reasoning tasks. We analyse our results and identify failure modes that can
serve as useful guides for future research, potentially informing strategies to
address the limitations observed in LLMs.
CLoVe: Encoding Compositional Language in Contrastive Vision-Language
  Models
Authors: name: Santiago Castro, name: Amir Ziai, name: Avneesh Saluja, name: Zhuoning Yuan, name: Rada Mihalcea
Published: 2024-02-22 23:42:25

Recent years have witnessed a significant increase in the performance of
Vision and Language tasks. Foundational Vision-Language Models (VLMs), such as
CLIP, have been leveraged in multiple settings and demonstrated remarkable
performance across several tasks. Such models excel at object-centric
recognition yet learn text representations that seem invariant to word order,
failing to compose known concepts in novel ways. However, no evidence exists
that any VLM, including large-scale single-stream models such as GPT-4V,
identifies compositions successfully. In this paper, we introduce a framework
to significantly improve the ability of existing models to encode compositional
language, with over 10% absolute improvement on compositionality benchmarks,
while maintaining or improving the performance on standard object-recognition
and retrieval benchmarks. Our code and pre-trained models are publicly
available at https://github.com/netflix/clove.
Orca-Math: Unlocking the potential of SLMs in Grade School Math
Authors: name: Arindam Mitra, name: Hamed Khanpour, name: Corby Rosset, name: Ahmed Awadallah
Published: 2024-02-16 23:44:38

Mathematical word problem-solving has long been recognized as a complex task
for small language models (SLMs). A recent study hypothesized that the smallest
model size, needed to achieve over 80% accuracy on the GSM8K benchmark, is 34
billion parameters. To reach this level of performance with smaller models,
researcher often train SLMs to generate Python code or use tools to help avoid
calculation errors. Additionally, they employ ensembling, where outputs of up
to 100 model runs are combined to arrive at a more accurate result. Result
selection is done using consensus, majority vote or a separate a verifier model
used in conjunction with the SLM. Ensembling provides a substantial boost in
accuracy but at a significant cost increase with multiple calls to the model
(e.g., Phi-GSM uses top-48 to boost the performance from 68.2 to 81.5).
  In this work, we present Orca-Math, a 7-billion-parameter SLM based on the
Mistral-7B, which achieves 86.81% on GSM8k without the need for multiple model
calls or the use of verifiers, code execution or any other external tools. Our
approach has the following key elements: (1) A high quality synthetic dataset
of 200K math problems created using a multi-agent setup where agents
collaborate to create the data, (2) An iterative learning techniques that
enables the SLM to practice solving problems, receive feedback on its solutions
and learn from preference pairs incorporating the SLM solutions and the
feedback. When trained with Supervised Fine-Tuning alone, Orca-Math achieves
81.50% on GSM8k pass@1 metric. With iterative preference learning, Orca-Math
achieves 86.81% pass@1. Orca-Math surpasses the performance of significantly
larger models such as LLAMA-2-70B, WizardMath-70B, Gemini-Pro, ChatGPT-3.5. It
also significantly outperforms other smaller models while using much smaller
data (hundreds of thousands vs. millions of problems).