Orionnnn/hub-stats huggingface dataset:
SUTRA: Scalable Multilingual Language Model Architecture
Authors: name: Abhijit Bendale, name: Michael Sapienza, name: Steven Ripplinger, name: Simon Gibbs, name: Jaewon Lee, name: Pranav Mistry
Published: 2024-05-07 20:11:44

In this paper, we introduce SUTRA, multilingual Large Language Model
architecture capable of understanding, reasoning, and generating text in over
50 languages. SUTRA's design uniquely decouples core conceptual understanding
from language-specific processing, which facilitates scalable and efficient
multilingual alignment and learning. Employing a Mixture of Experts framework
both in language and concept processing, SUTRA demonstrates both computational
efficiency and responsiveness. Through extensive evaluations, SUTRA is
demonstrated to surpass existing models like GPT-3.5, Llama2 by 20-30% on
leading Massive Multitask Language Understanding (MMLU) benchmarks for
multilingual tasks. SUTRA models are also online LLMs that can use knowledge
from the internet to provide hallucination-free, factual and up-to-date
responses while retaining their multilingual capabilities. Furthermore, we
explore the broader implications of its architecture for the future of
multilingual AI, highlighting its potential to democratize access to AI
technology globally and to improve the equity and utility of AI in regions with
predominantly non-English languages. Our findings suggest that SUTRA not only
fills pivotal gaps in multilingual model capabilities but also establishes a
new benchmark for operational efficiency and scalability in AI applications.
Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large
  Language Models in Code Generation from Scientific Plots
Authors: name: Chengyue Wu, name: Yixiao Ge, name: Qiushan Guo, name: Jiahao Wang, name: Zhixuan Liang, name: Zeyu Lu, name: Ying Shan, name: Ping Luo
Published: 2024-05-13 17:59:22

The remarkable progress of Multi-modal Large Language Models (MLLMs) has
attracted significant attention due to their superior performance in visual
contexts. However, their capabilities in turning visual figure to executable
code, have not been evaluated thoroughly. To address this, we introduce
Plot2Code, a comprehensive visual coding benchmark designed for a fair and
in-depth assessment of MLLMs. We carefully collect 132 manually selected
high-quality matplotlib plots across six plot types from publicly available
matplotlib galleries. For each plot, we carefully offer its source code, and an
descriptive instruction summarized by GPT-4. This approach enables Plot2Code to
extensively evaluate MLLMs' code capabilities across various input modalities.
Furthermore, we propose three automatic evaluation metrics, including code pass
rate, text-match ratio, and GPT-4V overall rating, for a fine-grained
assessment of the output code and rendered images. Instead of simply judging
pass or fail, we employ GPT-4V to make an overall judgement between the
generated and reference images, which has been shown to be consistent with
human evaluation. The evaluation results, which include analyses of 14 MLLMs
such as the proprietary GPT-4V, Gemini-Pro, and the open-sourced Mini-Gemini,
highlight the substantial challenges presented by Plot2Code. With Plot2Code, we
reveal that most existing MLLMs struggle with visual coding for text-dense
plots, heavily relying on textual instruction. We hope that the evaluation
results from Plot2Code on visual coding will guide the future development of
MLLMs. All data involved with Plot2Code are available at
https://huggingface.co/datasets/TencentARC/Plot2Code.
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and
  Composition of Experts
Authors: name: Raghu Prabhakar, name: Ram Sivaramakrishnan, name: Darshan Gandhi, name: Yun Du, name: Mingran Wang, name: Xiangyu Song, name: Kejie Zhang, name: Tianren Gao, name: Angela Wang, name: Karen Li, name: Yongning Sheng, name: Joshua Brot, name: Denis Sokolov, name: Apurv Vivek, name: Calvin Leung, name: Arjun Sabnis, name: Jiayu Bai, name: Tuowen Zhao, name: Mark Gottscho, name: David Jackson, name: Mark Luttrell, name: Manish K. Shah, name: Edison Chen, name: Kaizhao Liang, name: Swayambhoo Jain, name: Urmish Thakker, name: Dawei Huang, name: Sumti Jairath, name: Kevin J. Brown, name: Kunle Olukotun
Published: 2024-05-13 07:32:45

Monolithic large language models (LLMs) like GPT-4 have paved the way for
modern generative AI applications. Training, serving, and maintaining
monolithic LLMs at scale, however, remains prohibitively expensive and
challenging. The disproportionate increase in compute-to-memory ratio of modern
AI accelerators have created a memory wall, necessitating new methods to deploy
AI. Composition of Experts (CoE) is an alternative modular approach that lowers
the cost and complexity of training and serving. However, this approach
presents two key challenges when using conventional hardware: (1) without fused
operations, smaller models have lower operational intensity, which makes high
utilization more challenging to achieve; and (2) hosting a large number of
models can be either prohibitively expensive or slow when dynamically switching
between them.
  In this paper, we describe how combining CoE, streaming dataflow, and a
three-tier memory system scales the AI memory wall. We describe Samba-CoE, a
CoE system with 150 experts and a trillion total parameters. We deploy
Samba-CoE on the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) - a
commercial dataflow accelerator architecture that has been co-designed for
enterprise inference and training applications. The chip introduces a new
three-tier memory system with on-chip distributed SRAM, on-package HBM, and
off-package DDR DRAM. A dedicated inter-RDU network enables scaling up and out
over multiple sockets. We demonstrate speedups ranging from 2x to 13x on
various benchmarks running on eight RDU sockets compared with an unfused
baseline. We show that for CoE inference deployments, the 8-socket RDU Node
reduces machine footprint by up to 19x, speeds up model switching time by 15x
to 31x, and achieves an overall speedup of 3.7x over a DGX H100 and 6.6x over a
DGX A100.
Customizing Text-to-Image Models with a Single Image Pair
Authors: name: Maxwell Jones, name: Sheng-Yu Wang, name: Nupur Kumari, name: David Bau, name: Jun-Yan Zhu
Published: 2024-05-02 17:59:52

Art reinterpretation is the practice of creating a variation of a reference
work, making a paired artwork that exhibits a distinct artistic style. We ask
if such an image pair can be used to customize a generative model to capture
the demonstrated stylistic difference. We propose Pair Customization, a new
customization method that learns stylistic difference from a single image pair
and then applies the acquired style to the generation process. Unlike existing
methods that learn to mimic a single concept from a collection of images, our
method captures the stylistic difference between paired images. This allows us
to apply a stylistic change without overfitting to the specific image content
in the examples. To address this new task, we employ a joint optimization
method that explicitly separates the style and content into distinct LoRA
weight spaces. We optimize these style and content weights to reproduce the
style and content images while encouraging their orthogonality. During
inference, we modify the diffusion process via a new style guidance based on
our learned weights. Both qualitative and quantitative experiments show that
our method can effectively learn style while avoiding overfitting to image
content, highlighting the potential of modeling such stylistic differences from
a single image pair.
FLAME: Factuality-Aware Alignment for Large Language Models
Authors: name: Sheng-Chieh Lin, name: Luyu Gao, name: Barlas Oguz, name: Wenhan Xiong, name: Jimmy Lin, name: Wen-tau Yih, name: Xilun Chen
Published: 2024-05-02 17:54:54

Alignment is a standard procedure to fine-tune pre-trained large language
models (LLMs) to follow natural language instructions and serve as helpful AI
assistants. We have observed, however, that the conventional alignment process
fails to enhance the factual accuracy of LLMs, and often leads to the
generation of more false facts (i.e. hallucination). In this paper, we study
how to make the LLM alignment process more factual, by first identifying
factors that lead to hallucination in both alignment steps:\ supervised
fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that
training the LLM on new knowledge or unfamiliar texts can encourage
hallucination. This makes SFT less factual as it trains on human labeled data
that may be novel to the LLM. Furthermore, reward functions used in standard RL
can also encourage hallucination, because it guides the LLM to provide more
helpful responses on a diverse set of instructions, often preferring longer and
more detailed responses. Based on these observations, we propose
factuality-aware alignment, comprised of factuality-aware SFT and
factuality-aware RL through direct preference optimization. Experiments show
that our proposed factuality-aware alignment guides LLMs to output more factual
responses while maintaining instruction-following capability.
NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Authors: name: Gerald Shen, name: Zhilin Wang, name: Olivier Delalleau, name: Jiaqi Zeng, name: Yi Dong, name: Daniel Egert, name: Shengyang Sun, name: Jimmy Zhang, name: Sahil Jain, name: Ali Taghibakhshi, name: Markel Sanz Ausin, name: Ashwath Aithal, name: Oleksii Kuchaiev
Published: 2024-05-02 17:13:40

Aligning Large Language Models (LLMs) with human values and preferences is
essential for making them helpful and safe. However, building efficient tools
to perform alignment can be challenging, especially for the largest and most
competent LLMs which often contain tens or hundreds of billions of parameters.
We create NeMo-Aligner, a toolkit for model alignment that can efficiently
scale to using hundreds of GPUs for training. NeMo-Aligner comes with highly
optimized and scalable implementations for major paradigms of model alignment
such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference
Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally,
our toolkit supports running most of the alignment techniques in a Parameter
Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for
extensibility, allowing support for other alignment techniques with minimal
effort. It is open-sourced with Apache 2.0 License and we invite community
contributions at https://github.com/NVIDIA/NeMo-Aligner
Prometheus 2: An Open Source Language Model Specialized in Evaluating
  Other Language Models
Authors: name: Seungone Kim, name: Juyoung Suk, name: Shayne Longpre, name: Bill Yuchen Lin, name: Jamin Shin, name: Sean Welleck, name: Graham Neubig, name: Moontae Lee, name: Kyungjae Lee, name: Minjoon Seo
Published: 2024-05-02 17:59:35

Proprietary LMs such as GPT-4 are often employed to assess the quality of
responses from various LMs. However, concerns including transparency,
controllability, and affordability strongly motivate the development of
open-source LMs specialized in evaluations. On the other hand, existing open
evaluator LMs exhibit critical shortcomings: 1) they issue scores that
significantly diverge from those assigned by humans, and 2) they lack the
flexibility to perform both direct assessment and pairwise ranking, the two
most prevalent forms of assessment. Additionally, they do not possess the
ability to evaluate based on custom evaluation criteria, focusing instead on
general attributes like helpfulness and harmlessness. To address these issues,
we introduce Prometheus 2, a more powerful evaluator LM than its predecessor
that closely mirrors human and GPT-4 judgements. Moreover, it is capable of
processing both direct assessment and pair-wise ranking formats grouped with a
user-defined evaluation criteria. On four direct assessment benchmarks and four
pairwise ranking benchmarks, Prometheus 2 scores the highest correlation and
agreement with humans and proprietary LM judges among all tested open evaluator
LMs. Our models, code, and data are all publicly available at
https://github.com/prometheus-eval/prometheus-eval.
LLM-AD: Large Language Model based Audio Description System
Authors: name: Peng Chu, name: Jiang Wang, name: Andre Abrantes
Published: 2024-05-02 03:38:58

The development of Audio Description (AD) has been a pivotal step forward in
making video content more accessible and inclusive. Traditionally, AD
production has demanded a considerable amount of skilled labor, while existing
automated approaches still necessitate extensive training to integrate
multimodal inputs and tailor the output from a captioning style to an AD style.
In this paper, we introduce an automated AD generation pipeline that harnesses
the potent multimodal and instruction-following capacities of GPT-4V(ision).
Notably, our methodology employs readily available components, eliminating the
need for additional training. It produces ADs that not only comply with
established natural language AD production standards but also maintain
contextually consistent character information across frames, courtesy of a
tracking-based character recognition module. A thorough analysis on the MAD
dataset reveals that our approach achieves a performance on par with
learning-based methods in automated AD production, as substantiated by a CIDEr
score of 20.5.
WildChat: 1M ChatGPT Interaction Logs in the Wild
Authors: name: Wenting Zhao, name: Xiang Ren, name: Jack Hessel, name: Claire Cardie, name: Yejin Choi, name: Yuntian Deng
Published: 2024-05-02 17:00:02

Chatbots such as GPT-4 and ChatGPT are now serving millions of users. Despite
their widespread use, there remains a lack of public datasets showcasing how
these tools are used by a population of users in practice. To bridge this gap,
we offered free access to ChatGPT for online users in exchange for their
affirmative, consensual opt-in to anonymously collect their chat transcripts
and request headers. From this, we compiled WildChat, a corpus of 1 million
user-ChatGPT conversations, which consists of over 2.5 million interaction
turns. We compare WildChat with other popular user-chatbot interaction
datasets, and find that our dataset offers the most diverse user prompts,
contains the largest number of languages, and presents the richest variety of
potentially toxic use-cases for researchers to study. In addition to
timestamped chat transcripts, we enrich the dataset with demographic data,
including state, country, and hashed IP addresses, alongside request headers.
This augmentation allows for more detailed analysis of user behaviors across
different geographical regions and temporal dimensions. Finally, because it
captures a broad range of use cases, we demonstrate the dataset's potential
utility in fine-tuning instruction-following models. WildChat is released at
https://wildchat.allen.ai under AI2 ImpACT Licenses.
LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report
Authors: name: Justin Zhao, name: Timothy Wang, name: Wael Abid, name: Geoffrey Angus, name: Arnav Garg, name: Jeffery Kinnison, name: Alex Sherstinsky, name: Piero Molino, name: Travis Addair, name: Devvret Rishi
Published: 2024-04-29 04:01:45

Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted
methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models
(LLMs). LoRA reduces the number of trainable parameters and memory usage while
achieving comparable performance to full fine-tuning. We aim to assess the
viability of training and serving LLMs fine-tuned with LoRA in real-world
applications. First, we measure the quality of LLMs fine-tuned with quantized
low rank adapters across 10 base models and 31 tasks for a total of 310 models.
We find that 4-bit LoRA fine-tuned models outperform base models by 34 points
and GPT-4 by 10 points on average. Second, we investigate the most effective
base models for fine-tuning and assess the correlative and predictive
capacities of task complexity heuristics in forecasting the outcomes of
fine-tuning. Finally, we evaluate the latency and concurrency capabilities of
LoRAX, an open-source Multi-LoRA inference server that facilitates the
deployment of multiple LoRA fine-tuned models on a single GPU using shared base
model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web
application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA
A100 GPU with 80GB memory. LoRA Land highlights the quality and
cost-effectiveness of employing multiple specialized LLMs over a single,
general-purpose LLM.
StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video
  Generation
Authors: name: Yupeng Zhou, name: Daquan Zhou, name: Ming-Ming Cheng, name: Jiashi Feng, name: Qibin Hou
Published: 2024-05-02 16:25:16

For recent diffusion-based generative models, maintaining consistent content
across a series of generated images, especially those containing subjects and
complex details, presents a significant challenge. In this paper, we propose a
new way of self-attention calculation, termed Consistent Self-Attention, that
significantly boosts the consistency between the generated images and augments
prevalent pretrained diffusion-based text-to-image models in a zero-shot
manner. To extend our method to long-range video generation, we further
introduce a novel semantic space temporal motion prediction module, named
Semantic Motion Predictor. It is trained to estimate the motion conditions
between two provided images in the semantic spaces. This module converts the
generated sequence of images into videos with smooth transitions and consistent
subjects that are significantly more stable than the modules based on latent
spaces only, especially in the context of long video generation. By merging
these two novel components, our framework, referred to as StoryDiffusion, can
describe a text-based story with consistent images or videos encompassing a
rich variety of contents. The proposed StoryDiffusion encompasses pioneering
explorations in visual story generation with the presentation of images and
videos, which we hope could inspire more research from the aspect of
architectural modifications. Our code is made publicly available at
https://github.com/HVision-NKU/StoryDiffusion.
STT: Stateful Tracking with Transformers for Autonomous Driving
Authors: name: Longlong Jing, name: Ruichi Yu, name: Xu Chen, name: Zhengli Zhao, name: Shiwei Sheng, name: Colin Graber, name: Qi Chen, name: Qinru Li, name: Shangxuan Wu, name: Han Deng, name: Sangjin Lee, name: Chris Sweeney, name: Qiurui He, name: Wei-Chih Hung, name: Tong He, name: Xingyi Zhou, name: Farshid Moussavi, name: Zijian Guo, name: Yin Zhou, name: Mingxing Tan, name: Weilong Yang, name: Congcong Li
Published: 2024-04-30 23:04:36

Tracking objects in three-dimensional space is critical for autonomous
driving. To ensure safety while driving, the tracker must be able to reliably
track objects across frames and accurately estimate their states such as
velocity and acceleration in the present. Existing works frequently focus on
the association task while either neglecting the model performance on state
estimation or deploying complex heuristics to predict the states. In this
paper, we propose STT, a Stateful Tracking model built with Transformers, that
can consistently track objects in the scenes while also predicting their states
accurately. STT consumes rich appearance, geometry, and motion signals through
long term history of detections and is jointly optimized for both data
association and state estimation tasks. Since the standard tracking metrics
like MOTA and MOTP do not capture the combined performance of the two tasks in
the wider spectrum of object states, we extend them with new metrics called
S-MOTA and MOTPS that address this limitation. STT achieves competitive
real-time performance on the Waymo Open Dataset.
Automatic Creative Selection with Cross-Modal Matching
Authors: name: Alex Kim, name: Jia Huang, name: Rob Monarch, name: Jerry Kwac, name: Anikesh Kamath, name: Parmeshwar Khurd, name: Kailash Thiyagarajan, name: Goodman Gu
Published: 2024-02-28 22:05:38

Application developers advertise their Apps by creating product pages with
App images, and bidding on search terms. It is then crucial for App images to
be highly relevant with the search terms. Solutions to this problem require an
image-text matching model to predict the quality of the match between the
chosen image and the search terms. In this work, we present a novel approach to
matching an App image to search terms based on fine-tuning a pre-trained LXMERT
model. We show that compared to the CLIP model and a baseline using a
Transformer model for search terms, and a ResNet model for images, we
significantly improve the matching accuracy. We evaluate our approach using two
sets of labels: advertiser associated (image, search term) pairs for a given
application, and human ratings for the relevance between (image, search term)
pairs. Our approach achieves 0.96 AUC score for advertiser associated ground
truth, outperforming the transformer+ResNet baseline and the fine-tuned CLIP
model by 8% and 14%. For human labeled ground truth, our approach achieves 0.95
AUC score, outperforming the transformer+ResNet baseline and the fine-tuned
CLIP model by 16% and 17%.
Paint by Inpaint: Learning to Add Image Objects by Removing Them First
Authors: name: Navve Wasserman, name: Noam Rotstein, name: Roy Ganz, name: Ron Kimmel
Published: 2024-04-28 15:07:53

Image editing has advanced significantly with the introduction of
text-conditioned diffusion models. Despite this progress, seamlessly adding
objects to images based on textual instructions without requiring user-provided
input masks remains a challenge. We address this by leveraging the insight that
removing objects (Inpaint) is significantly simpler than its inverse process of
adding them (Paint), attributed to the utilization of segmentation mask
datasets alongside inpainting models that inpaint within these masks.
Capitalizing on this realization, by implementing an automated and extensive
pipeline, we curate a filtered large-scale image dataset containing pairs of
images and their corresponding object-removed versions. Using these pairs, we
train a diffusion model to inverse the inpainting process, effectively adding
objects into images. Unlike other editing datasets, ours features natural
target images instead of synthetic ones; moreover, it maintains consistency
between source and target by construction. Additionally, we utilize a large
Vision-Language Model to provide detailed descriptions of the removed objects
and a Large Language Model to convert these descriptions into diverse,
natural-language instructions. We show that the trained model surpasses
existing ones both qualitatively and quantitatively, and release the
large-scale dataset alongside the trained models for the community.