Orionnnn/hub-stats huggingface dataset:
UFO: A UI-Focused Agent for Windows OS Interaction
Authors: name: Chaoyun Zhang, name: Liqun Li, name: Shilin He, name: Xu Zhang, name: Bo Qiao, name: Si Qin, name: Minghua Ma, name: Yu Kang, name: Qingwei Lin, name: Saravan Rajmohan, name: Dongmei Zhang, name: Qi Zhang
Published: 2024-02-08 15:40:35

We introduce UFO, an innovative UI-Focused agent to fulfill user requests
tailored to applications on Windows OS, harnessing the capabilities of
GPT-Vision. UFO employs a dual-agent framework to meticulously observe and
analyze the graphical user interface (GUI) and control information of Windows
applications. This enables the agent to seamlessly navigate and operate within
individual applications and across them to fulfill user requests, even when
spanning multiple applications. The framework incorporates a control
interaction module, facilitating action grounding without human intervention
and enabling fully automated execution. Consequently, UFO transforms arduous
and time-consuming processes into simple tasks achievable solely through
natural language commands. We conducted testing of UFO across 9 popular Windows
applications, encompassing a variety of scenarios reflective of users' daily
usage. The results, derived from both quantitative metrics and real-case
studies, underscore the superior effectiveness of UFO in fulfilling user
requests. To the best of our knowledge, UFO stands as the first UI agent
specifically tailored for task completion within the Windows OS environment.
The open-source code for UFO is available on https://github.com/microsoft/UFO.
Vision-Based Hand Gesture Customization from a Single Demonstration
Authors: name: Soroush Shahi, name: Cori Tymoszek Park, name: Richard Kang, name: Asaf Liberman, name: Oron Levy, name: Jun Gong, name: Abdelkareem Bedri, name: Gierad Laput
Published: 2024-02-13 12:49:13

Hand gesture recognition is becoming a more prevalent mode of human-computer
interaction, especially as cameras proliferate across everyday devices. Despite
continued progress in this field, gesture customization is often underexplored.
Customization is crucial since it enables users to define and demonstrate
gestures that are more natural, memorable, and accessible. However,
customization requires efficient usage of user-provided data. We introduce a
method that enables users to easily design bespoke gestures with a monocular
camera from one demonstration. We employ transformers and meta-learning
techniques to address few-shot learning challenges. Unlike prior work, our
method supports any combination of one-handed, two-handed, static, and dynamic
gestures, including different viewpoints. We evaluated our customization method
through a user study with 20 gestures collected from 21 participants, achieving
up to 97% average recognition accuracy from one demonstration. Our work
provides a viable path for vision-based gesture customization, laying the
foundation for future advancements in this domain.
NeRF Analogies: Example-Based Visual Attribute Transfer for NeRFs
Authors: name: Michael Fischer, name: Zhengqin Li, name: Thu Nguyen-Phuoc, name: Aljaz Bozic, name: Zhao Dong, name: Carl Marshall, name: Tobias Ritschel
Published: 2024-02-13 17:47:42

A Neural Radiance Field (NeRF) encodes the specific relation of 3D geometry
and appearance of a scene. We here ask the question whether we can transfer the
appearance from a source NeRF onto a target 3D geometry in a semantically
meaningful way, such that the resulting new NeRF retains the target geometry
but has an appearance that is an analogy to the source NeRF. To this end, we
generalize classic image analogies from 2D images to NeRFs. We leverage
correspondence transfer along semantic affinity that is driven by semantic
features from large, pre-trained 2D image models to achieve multi-view
consistent appearance transfer. Our method allows exploring the mix-and-match
product space of 3D geometry and appearance. We show that our method
outperforms traditional stylization-based methods and that a large majority of
users prefer our method over several typical baselines.
Mixtures of Experts Unlock Parameter Scaling for Deep RL
Authors: name: Johan Obando-Ceron, name: Ghada Sokar, name: Timon Willi, name: Clare Lyle, name: Jesse Farebrother, name: Jakob Foerster, name: Gintare Karolina Dziugaite, name: Doina Precup, name: Pablo Samuel Castro
Published: 2024-02-13 17:18:56

The recent rapid progress in (self) supervised learning models is in large
part predicted by empirical scaling laws: a model's performance scales
proportionally to its size. Analogous scaling laws remain elusive for
reinforcement learning domains, however, where increasing the parameter count
of a model often hurts its final performance. In this paper, we demonstrate
that incorporating Mixture-of-Expert (MoE) modules, and in particular Soft MoEs
(Puigcerver et al., 2023), into value-based networks results in more
parameter-scalable models, evidenced by substantial performance increases
across a variety of training regimes and model sizes. This work thus provides
strong empirical evidence towards developing scaling laws for reinforcement
learning.
Graph Mamba: Towards Learning on Graphs with State Space Models
Authors: name: Ali Behrouz, name: Farnoosh Hashemi
Published: 2024-02-13 18:58:17

Graph Neural Networks (GNNs) have shown promising potential in graph
representation learning. The majority of GNNs define a local message-passing
mechanism, propagating information over the graph by stacking multiple layers.
These methods, however, are known to suffer from two major limitations:
over-squashing and poor capturing of long-range dependencies. Recently, Graph
Transformers (GTs) emerged as a powerful alternative to Message-Passing Neural
Networks (MPNNs). GTs, however, have quadratic computational cost, lack
inductive biases on graph structures, and rely on complex Positional/Structural
Encodings (SE/PE). In this paper, we show that while Transformers, complex
message-passing, and SE/PE are sufficient for good performance in practice,
neither is necessary. Motivated by the recent success of State Space Models
(SSMs), such as Mamba, we present Graph Mamba Networks (GMNs), a general
framework for a new class of GNNs based on selective SSMs. We discuss and
categorize the new challenges when adopting SSMs to graph-structured data, and
present four required and one optional steps to design GMNs, where we choose
(1) Neighborhood Tokenization, (2) Token Ordering, (3) Architecture of
Bidirectional Selective SSM Encoder, (4) Local Encoding, and dispensable (5) PE
and SE. We further provide theoretical justification for the power of GMNs.
Experiments demonstrate that despite much less computational cost, GMNs attain
an outstanding performance in long-range, small-scale, large-scale, and
heterophilic benchmark datasets.
IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality
  3D Generation
Authors: name: Luke Melas-Kyriazi, name: Iro Laina, name: Christian Rupprecht, name: Natalia Neverova, name: Andrea Vedaldi, name: Oran Gafni, name: Filippos Kokkinos
Published: 2024-02-13 18:59:51

Most text-to-3D generators build upon off-the-shelf text-to-image models
trained on billions of images. They use variants of Score Distillation Sampling
(SDS), which is slow, somewhat unstable, and prone to artifacts. A mitigation
is to fine-tune the 2D generator to be multi-view aware, which can help
distillation or can be combined with reconstruction networks to output 3D
objects directly. In this paper, we further explore the design space of
text-to-3D models. We significantly improve multi-view generation by
considering video instead of image generators. Combined with a 3D
reconstruction algorithm which, by using Gaussian splatting, can optimize a
robust image-based loss, we directly produce high-quality 3D outputs from the
generated views. Our new method, IM-3D, reduces the number of evaluations of
the 2D generator network 10-100x, resulting in a much more efficient pipeline,
better quality, fewer geometric inconsistencies, and higher yield of usable 3D
assets.
Tandem Transformers for Inference Efficient LLMs
Authors: name: Aishwarya P S, name: Pranav Ajit Nair, name: Yashas Samaga, name: Toby Boyd, name: Sanjiv Kumar, name: Prateek Jain, name: Praneeth Netrapalli
Published: 2024-02-13 18:24:08

The autoregressive nature of conventional large language models (LLMs)
inherently limits inference speed, as tokens are generated sequentially. While
speculative and parallel decoding techniques attempt to mitigate this, they
face limitations: either relying on less accurate smaller models for generation
or failing to fully leverage the base LLM's representations.
  We introduce a novel architecture, Tandem transformers, to address these
issues. This architecture uniquely combines (1) a small autoregressive model
and (2) a large model operating in block mode (processing multiple tokens
simultaneously). The small model's predictive accuracy is substantially
enhanced by granting it attention to the large model's richer representations.
On the PaLM2 pretraining dataset, a tandem of PaLM2-Bison and PaLM2-Gecko
demonstrates a 3.3% improvement in next-token prediction accuracy over a
standalone PaLM2-Gecko, offering a 1.16x speedup compared to a PaLM2-Otter
model with comparable downstream performance. We further incorporate the tandem
model within the speculative decoding (SPEED) framework where the large model
validates tokens from the small model. This ensures that the Tandem of
PaLM2-Bison and PaLM2-Gecko achieves substantial speedup (around 1.14x faster
than using vanilla PaLM2-Gecko in SPEED) while maintaining identical downstream
task accuracy.
World Model on Million-Length Video And Language With RingAttention
Authors: name: Hao Liu, name: Wilson Yan, name: Matei Zaharia, name: Pieter Abbeel
Published: 2024-02-13 07:47:36

Current language models fall short in understanding aspects of the world not
easily described in words, and struggle with complex, long-form tasks. Video
sequences offer valuable temporal information absent in language and static
images, making them attractive for joint modeling with language. Such models
could develop a understanding of both human textual knowledge and the physical
world, enabling broader AI capabilities for assisting humans. However, learning
from millions of tokens of video and language sequences poses challenges due to
memory constraints, computational complexity, and limited datasets. To address
these challenges, we curate a large dataset of diverse videos and books,
utilize the RingAttention technique to scalably train on long sequences, and
gradually increase context size from 4K to 1M tokens. This paper makes the
following contributions: (a) Largest context size neural network: We train one
of the largest context size transformers on long video and language sequences,
setting new benchmarks in difficult retrieval tasks and long video
understanding. (b) Solutions for overcoming vision-language training
challenges, including using masked sequence packing for mixing different
sequence lengths, loss weighting to balance language and vision, and
model-generated QA dataset for long sequence chat. (c) A highly-optimized
implementation with RingAttention, masked sequence packing, and other key
features for training on millions-length multimodal sequences. (d) Fully
open-sourced a family of 7B parameter models capable of processing long text
documents (LWM-Text, LWM-Text-Chat) and videos (LWM, LWM-Chat) of over 1M
tokens. This work paves the way for training on massive datasets of long video
and language to develop understanding of both human knowledge and the
multimodal world, and broader capabilities.
BASE TTS: Lessons from building a billion-parameter Text-to-Speech model
  on 100K hours of data
Authors: name: Mateusz Łajszczak, name: Guillermo Cámbara, name: Yang Li, name: Fatih Beyhan, name: Arent van Korlaar, name: Fan Yang, name: Arnaud Joly, name: Álvaro Martín-Cortinas, name: Ammar Abbas, name: Adam Michalski, name: Alexis Moinet, name: Sri Karlapati, name: Ewa Muszyńska, name: Haohan Guo, name: Bartosz Putrycz, name: Soledad López Gambino, name: Kayeon Yoo, name: Elena Sokolova, name: Thomas Drugman
Published: 2024-02-12 22:21:30

We introduce a text-to-speech (TTS) model called BASE TTS, which stands for
Big Adaptive Streamable TTS with
Emergent abilities. BASE TTS is the largest TTS model to-date,
trained on 100K hours of public domain speech data, achieving a new
state-of-the-art in speech naturalness. It deploys a 1-billion-parameter
autoregressive Transformer that converts raw texts into discrete codes
("speechcodes") followed by a convolution-based decoder which converts these
speechcodes into waveforms in an incremental, streamable manner. Further, our
speechcodes are built using a novel speech tokenization technique that features
speaker ID disentanglement and compression with byte-pair encoding. Echoing the
widely-reported "emergent abilities" of large language models when trained on
increasing volume of data, we show that BASE TTS variants built with 10K+ hours
and 500M+ parameters begin to demonstrate natural prosody on textually complex
sentences. We design and share a specialized dataset to measure these emergent
abilities for text-to-speech. We showcase state-of-the-art naturalness of BASE
TTS by evaluating against baselines that include publicly available large-scale
text-to-speech systems: YourTTS, Bark and TortoiseTTS. Audio samples generated
by the model can be heard at https://amazon-ltts-paper.com/.
Lumos : Empowering Multimodal LLMs with Scene Text Recognition
Authors: name: Ashish Shenoy, name: Yichao Lu, name: Srihari Jayakumar, name: Debojeet Chatterjee, name: Mohsen Moslehpour, name: Pierce Chuang, name: Abhay Harpale, name: Vikas Bhardwaj, name: Di Xu, name: Shicong Zhao, name: Longfang Zhao, name: Ankit Ramchandani, name: Xin Luna Dong, name: Anuj Kumar
Published: 2024-02-12 19:27:26

We introduce Lumos, the first end-to-end multimodal question-answering system
with text understanding capabilities. At the core of Lumos is a Scene Text
Recognition (STR) component that extracts text from first person point-of-view
images, the output of which is used to augment input to a Multimodal Large
Language Model (MM-LLM). While building Lumos, we encountered numerous
challenges related to STR quality, overall latency, and model inference. In
this paper, we delve into those challenges, and discuss the system
architecture, design choices, and modeling techniques employed to overcome
these obstacles. We also provide a comprehensive evaluation for each component,
showcasing high quality and efficiency.
Learning Continuous 3D Words for Text-to-Image Generation
Authors: name: Ta-Ying Cheng, name: Matheus Gadelha, name: Thibault Groueix, name: Matthew Fisher, name: Radomir Mech, name: Andrew Markham, name: Niki Trigoni
Published: 2024-02-13 18:34:10

Current controls over diffusion models (e.g., through text or ControlNet) for
image generation fall short in recognizing abstract, continuous attributes like
illumination direction or non-rigid shape change. In this paper, we present an
approach for allowing users of text-to-image models to have fine-grained
control of several attributes in an image. We do this by engineering special
sets of input tokens that can be transformed in a continuous manner -- we call
them Continuous 3D Words. These attributes can, for example, be represented as
sliders and applied jointly with text prompts for fine-grained control over
image generation. Given only a single mesh and a rendering engine, we show that
our approach can be adopted to provide continuous user control over several
3D-aware attributes, including time-of-day illumination, bird wing orientation,
dollyzoom effect, and object poses. Our method is capable of conditioning image
creation with multiple Continuous 3D Words and text descriptions simultaneously
while adding no overhead to the generative process. Project Page:
https://ttchengab.github.io/continuous_3d_words
ODIN: Disentangled Reward Mitigates Hacking in RLHF
Authors: name: Lichang Chen, name: Chen Zhu, name: Davit Soselia, name: Jiuhai Chen, name: Tianyi Zhou, name: Tom Goldstein, name: Heng Huang, name: Mohammad Shoeybi, name: Bryan Catanzaro
Published: 2024-02-11 22:40:12

In this work, we study the issue of reward hacking on the response length, a
challenge emerging in Reinforcement Learning from Human Feedback (RLHF) on
LLMs. A well-formatted, verbose but less helpful response from the LLMs can
often deceive LLMs or even human evaluators to achieve high scores. The same
issue also holds for some reward models in RL. To address the challenges in
both training and evaluation, we establish a more reliable evaluation protocol
for comparing different training configurations, which inspects the trade-off
between LLM evaluation score and response length obtained by varying training
hyperparameters. Based on this evaluation, we conduct large-scale studies,
where the results shed insights into the efficacy of hyperparameters and tricks
used in RL on mitigating length bias. We further propose to improve the reward
model by jointly training two linear heads on shared feature representations to
predict the rewards, one trained to correlate with length, and the other
trained to decorrelate with length and therefore focus more on the actual
content. We then discard the length head in RL to prevent reward hacking on
length. Experiments demonstrate that our approach almost eliminates the reward
correlation with length, and improves the obtained policy by a significant
margin.
Making Flow-Matching-Based Zero-Shot Text-to-Speech Laugh as You Like
Authors: name: Naoyuki Kanda, name: Xiaofei Wang, name: Sefik Emre Eskimez, name: Manthan Thakker, name: Hemin Yang, name: Zirun Zhu, name: Min Tang, name: Canrun Li, name: Steven Tsai, name: Zhen Xiao, name: Yufei Xia, name: Jinzhu Li, name: Yanqing Liu, name: Sheng Zhao, name: Michael Zeng
Published: 2024-02-12 02:58:10

Laughter is one of the most expressive and natural aspects of human speech,
conveying emotions, social cues, and humor. However, most text-to-speech (TTS)
systems lack the ability to produce realistic and appropriate laughter sounds,
limiting their applications and user experience. While there have been prior
works to generate natural laughter, they fell short in terms of controlling the
timing and variety of the laughter to be generated. In this work, we propose
ELaTE, a zero-shot TTS that can generate natural laughing speech of any speaker
based on a short audio prompt with precise control of laughter timing and
expression. Specifically, ELaTE works on the audio prompt to mimic the voice
characteristic, the text prompt to indicate the contents of the generated
speech, and the input to control the laughter expression, which can be either
the start and end times of laughter, or the additional audio prompt that
contains laughter to be mimicked. We develop our model based on the foundation
of conditional flow-matching-based zero-shot TTS, and fine-tune it with
frame-level representation from a laughter detector as additional conditioning.
With a simple scheme to mix small-scale laughter-conditioned data with
large-scale pre-training data, we demonstrate that a pre-trained zero-shot TTS
model can be readily fine-tuned to generate natural laughter with precise
controllability, without losing any quality of the pre-trained zero-shot TTS
model. Through the evaluations, we show that ELaTE can generate laughing speech
with significantly higher quality and controllability compared to conventional
models. See https://aka.ms/elate/ for demo samples.
LiRank: Industrial Large Scale Ranking Models at LinkedIn
Authors: name: Fedor Borisyuk, name: Mingzhou Zhou, name: Qingquan Song, name: Siyu Zhu, name: Birjodh Tiwana, name: Ganesh Parameswaran, name: Siddharth Dangi, name: Lars Hertel, name: Qiang Xiao, name: Xiaochen Hou, name: Yunbo Ouyang, name: Aman Gupta, name: Sheallika Singh, name: Dan Liu, name: Hailing Cheng, name: Lei Le, name: Jonathan Hung, name: Sathiya Keerthi, name: Ruoyan Wang, name: Fengyu Zhang, name: Mohit Kothari, name: Chen Zhu, name: Daqi Sun, name: Yun Dai, name: Xun Luan, name: Sirou Zhu, name: Zhiwei Wang, name: Neil Daftary, name: Qianqi Shen, name: Chengming Jiang, name: Haichao Wei, name: Maneesh Varshney, name: Amol Ghoting, name: Souvik Ghosh
Published: 2024-02-10 01:47:10

We present LiRank, a large-scale ranking framework at LinkedIn that brings to
production state-of-the-art modeling architectures and optimization methods. We
unveil several modeling improvements, including Residual DCN, which adds
attention and residual connections to the famous DCNv2 architecture. We share
insights into combining and tuning SOTA architectures to create a unified
model, including Dense Gating, Transformers and Residual DCN. We also propose
novel techniques for calibration and describe how we productionalized deep
learning based explore/exploit methods. To enable effective, production-grade
serving of large ranking models, we detail how to train and compress models
using quantization and vocabulary compression. We provide details about the
deployment setup for large-scale use cases of Feed ranking, Jobs
Recommendations, and Ads click-through rate (CTR) prediction. We summarize our
learnings from various A/B tests by elucidating the most effective technical
approaches. These ideas have contributed to relative metrics improvements
across the board at LinkedIn: +0.5% member sessions in the Feed, +1.76%
qualified job applications for Jobs search and recommendations, and +4.3% for
Ads CTR. We hope this work can provide practical insights and solutions for
practitioners interested in leveraging large-scale deep ranking systems.
Policy Improvement using Language Feedback Models
Authors: name: Victor Zhong, name: Dipendra Misra, name: Xingdi Yuan, name: Marc-Alexandre Côté
Published: 2024-02-12 18:41:34

We introduce Language Feedback Models (LFMs) that identify desirable
behaviour - actions that help achieve tasks specified in the instruction - for
imitation learning in instruction following. To train LFMs, we obtain feedback
from Large Language Models (LLMs) on visual trajectories verbalized to language
descriptions. First, by using LFMs to identify desirable behaviour to imitate,
we improve in task-completion rate over strong behavioural cloning baselines on
three distinct language grounding environments (Touchdown, ScienceWorld, and
ALFWorld). Second, LFMs outperform using LLMs as experts to directly predict
actions, when controlling for the number of LLM output tokens. Third, LFMs
generalize to unseen environments, improving task-completion rate by 3.5-12.0%
through one round of adaptation. Finally, LFM can be modified to provide
human-interpretable feedback without performance loss, allowing human
verification of desirable behaviour for imitation learning.