Orionnnn/hub-stats huggingface dataset:
Q-Instruct: Improving Low-level Visual Abilities for Multi-modality
  Foundation Models
Authors: name: Haoning Wu, name: Zicheng Zhang, name: Erli Zhang, name: Chaofeng Chen, name: Liang Liao, name: Annan Wang, name: Kaixin Xu, name: Chunyi Li, name: Jingwen Hou, name: Guangtao Zhai, name: Geng Xue, name: Wenxiu Sun, name: Qiong Yan, name: Weisi Lin
Published: 2023-11-12 09:10:51

Multi-modality foundation models, as represented by GPT-4V, have brought a
new paradigm for low-level visual perception and understanding tasks, that can
respond to a broad range of natural human instructions in a model. While
existing foundation models have shown exciting potentials on low-level visual
tasks, their related abilities are still preliminary and need to be improved.
In order to enhance these models, we conduct a large-scale subjective
experiment collecting a vast number of real human feedbacks on low-level
vision. Each feedback follows a pathway that starts with a detailed description
on the low-level visual appearance (*e.g. clarity, color, brightness* of an
image, and ends with an overall conclusion, with an average length of 45 words.
The constructed **Q-Pathway** dataset includes 58K detailed human feedbacks on
18,973 images with diverse low-level appearance. Moreover, to enable foundation
models to robustly respond to diverse types of questions, we design a
GPT-participated conversion to process these feedbacks into diverse-format 200K
instruction-response pairs. Experimental results indicate that the
**Q-Instruct** consistently elevates low-level perception and understanding
abilities across several foundational models. We anticipate that our datasets
can pave the way for a future that general intelligence can perceive,
understand low-level visual appearance and evaluate visual quality like a
human. Our dataset, model zoo, and demo is published at:
https://q-future.github.io/Q-Instruct.
Hiformer: Heterogeneous Feature Interactions Learning with Transformers
  for Recommender Systems
Authors: name: Huan Gui, name: Ruoxi Wang, name: Ke Yin, name: Long Jin, name: Maciej Kula, name: Taibai Xu, name: Lichan Hong, name: Ed H. Chi
Published: 2023-11-10 05:57:57

Learning feature interaction is the critical backbone to building recommender
systems. In web-scale applications, learning feature interaction is extremely
challenging due to the sparse and large input feature space; meanwhile,
manually crafting effective feature interactions is infeasible because of the
exponential solution space. We propose to leverage a Transformer-based
architecture with attention layers to automatically capture feature
interactions. Transformer architectures have witnessed great success in many
domains, such as natural language processing and computer vision. However,
there has not been much adoption of Transformer architecture for feature
interaction modeling in industry. We aim at closing the gap. We identify two
key challenges for applying the vanilla Transformer architecture to web-scale
recommender systems: (1) Transformer architecture fails to capture the
heterogeneous feature interactions in the self-attention layer; (2) The serving
latency of Transformer architecture might be too high to be deployed in
web-scale recommender systems. We first propose a heterogeneous self-attention
layer, which is a simple yet effective modification to the self-attention layer
in Transformer, to take into account the heterogeneity of feature interactions.
We then introduce Hiformer (Heterogeneous
Interaction Transformer) to further improve the model
expressiveness. With low-rank approximation and model pruning, \hiformer enjoys
fast inference for online deployment. Extensive offline experiment results
corroborates the effectiveness and efficiency of the Hiformer model.
We have successfully deployed the Hiformer model to a real world large
scale App ranking model at Google Play, with significant improvement in key
engagement metrics (up to +2.66\%).
FMViT: A multiple-frequency mixing Vision Transformer
Authors: name: Wei Tan, name: Yifeng Geng, name: Xuansong Xie
Published: 2023-11-09 19:33:50

The transformer model has gained widespread adoption in computer vision tasks
in recent times. However, due to the quadratic time and memory complexity of
self-attention, which is proportional to the number of input tokens, most
existing Vision Transformers (ViTs) encounter challenges in achieving efficient
performance in practical industrial deployment scenarios, such as TensorRT and
CoreML, where traditional CNNs excel. Although some recent attempts have been
made to design CNN-Transformer hybrid architectures to tackle this problem,
their overall performance has not met expectations. To tackle these challenges,
we propose an efficient hybrid ViT architecture named FMViT. This approach
enhances the model's expressive power by blending high-frequency features and
low-frequency features with varying frequencies, enabling it to capture both
local and global information effectively. Additionally, we introduce
deploy-friendly mechanisms such as Convolutional Multigroup Reparameterization
(gMLP), Lightweight Multi-head Self-Attention (RLMHSA), and Convolutional
Fusion Block (CFB) to further improve the model's performance and reduce
computational overhead. Our experiments demonstrate that FMViT surpasses
existing CNNs, ViTs, and CNNTransformer hybrid architectures in terms of
latency/accuracy trade-offs for various vision tasks. On the TensorRT platform,
FMViT outperforms Resnet101 by 2.5% (83.3% vs. 80.8%) in top-1 accuracy on the
ImageNet dataset while maintaining similar inference latency. Moreover, FMViT
achieves comparable performance with EfficientNet-B5, but with a 43%
improvement in inference speed. On CoreML, FMViT outperforms MobileOne by 2.6%
in top-1 accuracy on the ImageNet dataset, with inference latency comparable to
MobileOne (78.5% vs. 75.9%). Our code can be found at
https://github.com/tany0699/FMViT.
Florence-2: Advancing a Unified Representation for a Variety of Vision
  Tasks
Authors: name: Bin Xiao, name: Haiping Wu, name: Weijian Xu, name: Xiyang Dai, name: Houdong Hu, name: Yumao Lu, name: Michael Zeng, name: Ce Liu, name: Lu Yuan
Published: 2023-11-10 18:59:08

We introduce Florence-2, a novel vision foundation model with a unified,
prompt-based representation for a variety of computer vision and
vision-language tasks. While existing large vision models excel in transfer
learning, they struggle to perform a diversity of tasks with simple
instructions, a capability that implies handling the complexity of various
spatial hierarchy and semantic granularity. Florence-2 was designed to take
text-prompt as task instructions and generate desirable results in text forms,
whether it be captioning, object detection, grounding or segmentation. This
multi-task learning setup demands large-scale, high-quality annotated data. To
this end, we co-developed FLD-5B that consists of 5.4 billion comprehensive
visual annotations on 126 million images, using an iterative strategy of
automated image annotation and model refinement. We adopted a
sequence-to-sequence structure to train Florence-2 to perform versatile and
comprehensive vision tasks. Extensive evaluations on numerous tasks
demonstrated Florence-2 to be a strong vision foundation model contender with
unprecedented zero-shot and fine-tuning capabilities.
PolyMaX: General Dense Prediction with Mask Transformer
Authors: name: Xuan Yang, name: Liangzhe Yuan, name: Kimberly Wilber, name: Astuti Sharma, name: Xiuye Gu, name: Siyuan Qiao, name: Stephanie Debats, name: Huisheng Wang, name: Hartwig Adam, name: Mikhail Sirotenko, name: Liang-Chieh Chen
Published: 2023-11-09 22:29:21

Dense prediction tasks, such as semantic segmentation, depth estimation, and
surface normal prediction, can be easily formulated as per-pixel classification
(discrete outputs) or regression (continuous outputs). This per-pixel
prediction paradigm has remained popular due to the prevalence of fully
convolutional networks. However, on the recent frontier of segmentation task,
the community has been witnessing a shift of paradigm from per-pixel prediction
to cluster-prediction with the emergence of transformer architectures,
particularly the mask transformers, which directly predicts a label for a mask
instead of a pixel. Despite this shift, methods based on the per-pixel
prediction paradigm still dominate the benchmarks on the other dense prediction
tasks that require continuous outputs, such as depth estimation and surface
normal prediction. Motivated by the success of DORN and AdaBins in depth
estimation, achieved by discretizing the continuous output space, we propose to
generalize the cluster-prediction based method to general dense prediction
tasks. This allows us to unify dense prediction tasks with the mask transformer
framework. Remarkably, the resulting model PolyMaX demonstrates
state-of-the-art performance on three benchmarks of NYUD-v2 dataset. We hope
our simple yet effective design can inspire more research on exploiting mask
transformers for more dense prediction tasks. Code and model will be made
available.
Mirasol3B: A Multimodal Autoregressive model for time-aligned and
  contextual modalities
Authors: name: AJ Piergiovanni, name: Isaac Nobel, name: Dahun Kim, name: Michael S. Ryoo, name: Victor Gomes, name: Anelia Angelova
Published: 2023-11-09 19:15:12

One of the main challenges of multimodal learning is the need to combine
heterogeneous modalities (e.g., video, audio, text). For example, video and
audio are obtained at much higher rates than text and are roughly aligned in
time. They are often not synchronized with text, which comes as a global
context, e.g., a title, or a description. Furthermore, video and audio inputs
are of much larger volumes, and grow as the video length increases, which
naturally requires more compute dedicated to these modalities and makes
modeling of long-range dependencies harder.
  We here decouple the multimodal modeling, dividing it into separate, focused
autoregressive models, processing the inputs according to the characteristics
of the modalities. We propose a multimodal model, called Mirasol3B, consisting
of an autoregressive component for the time-synchronized modalities (audio and
video), and an autoregressive component for the context modalities which are
not necessarily aligned in time but are still sequential. To address the
long-sequences of the video-audio inputs, we propose to further partition the
video and audio sequences in consecutive snippets and autoregressively process
their representations. To that end, we propose a Combiner mechanism, which
models the audio-video information jointly within a timeframe. The Combiner
learns to extract audio and video features from raw spatio-temporal signals,
and then learns to fuse these features producing compact but expressive
representations per snippet.
  Our approach achieves the state-of-the-art on well established multimodal
benchmarks, outperforming much larger models. It effectively addresses the high
computational demand of media inputs by both learning compact representations,
controlling the sequence length of the audio-video feature representations, and
modeling their dependencies in time.
Instant3D: Fast Text-to-3D with Sparse-View Generation and Large
  Reconstruction Model
Authors: name: Jiahao Li, name: Hao Tan, name: Kai Zhang, name: Zexiang Xu, name: Fujun Luan, name: Yinghao Xu, name: Yicong Hong, name: Kalyan Sunkavalli, name: Greg Shakhnarovich, name: Sai Bi
Published: 2023-11-10 18:03:44

Text-to-3D with diffusion models have achieved remarkable progress in recent
years. However, existing methods either rely on score distillation-based
optimization which suffer from slow inference, low diversity and Janus
problems, or are feed-forward methods that generate low quality results due to
the scarcity of 3D training data. In this paper, we propose Instant3D, a novel
method that generates high-quality and diverse 3D assets from text prompts in a
feed-forward manner. We adopt a two-stage paradigm, which first generates a
sparse set of four structured and consistent views from text in one shot with a
fine-tuned 2D text-to-image diffusion model, and then directly regresses the
NeRF from the generated images with a novel transformer-based sparse-view
reconstructor. Through extensive experiments, we demonstrate that our method
can generate high-quality, diverse and Janus-free 3D assets within 20 seconds,
which is two order of magnitude faster than previous optimization-based methods
that can take 1 to 10 hours. Our project webpage: https://jiahao.ai/instant3d/.
FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor
  Cores
Authors: name: Daniel Y. Fu, name: Hermann Kumbong, name: Eric Nguyen, name: Christopher Ré
Published: 2023-11-10 07:33:35

Convolution models with long filters have demonstrated state-of-the-art
reasoning abilities in many long-sequence tasks but lag behind the most
optimized Transformers in wall-clock time. A major bottleneck is the Fast
Fourier Transform (FFT)--which allows long convolutions to run in O(N logN)
time in sequence length N but has poor hardware utilization. In this paper,
we study how to optimize the FFT convolution. We find two key bottlenecks: the
FFT does not effectively use specialized matrix multiply units, and it incurs
expensive I/O between layers of the memory hierarchy. In response, we propose
FlashFFTConv. FlashFFTConv uses a matrix decomposition that computes the FFT
using matrix multiply units and enables kernel fusion for long sequences,
reducing I/O. We also present two sparse convolution algorithms--1) partial
convolutions and 2) frequency-sparse convolutions--which can be implemented
simply by skipping blocks in the matrix decomposition, enabling further
opportunities for memory and compute savings. FlashFFTConv speeds up exact FFT
convolutions by up to 7.93times over PyTorch and achieves up to 4.4times
speedup end-to-end. Given the same compute budget, FlashFFTConv allows
Hyena-GPT-s to achieve 2.3 points better perplexity on the PILE and
M2-BERT-base to achieve 3.3 points higher GLUE score--matching models with
twice the parameter count. FlashFFTConv also achieves 96.1% accuracy on
Path-512, a high-resolution vision task where no model had previously achieved
better than 50%. Furthermore, partial convolutions enable longer-sequence
models--yielding the first DNA model that can process the longest human genes
(2.3M base pairs)--and frequency-sparse convolutions speed up pretrained models
while maintaining or improving model quality.
Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
Authors: name: Weiyang Liu, name: Zeju Qiu, name: Yao Feng, name: Yuliang Xiu, name: Yuxuan Xue, name: Longhui Yu, name: Haiwen Feng, name: Zhen Liu, name: Juyeon Heo, name: Songyou Peng, name: Yandong Wen, name: Michael J. Black, name: Adrian Weller, name: Bernhard Schölkopf
Published: 2023-11-10 18:59:54

Large foundation models are becoming ubiquitous, but training them from
scratch is prohibitively expensive. Thus, efficiently adapting these powerful
models to downstream tasks is increasingly important. In this paper, we study a
principled finetuning paradigm -- Orthogonal Finetuning (OFT) -- for downstream
task adaptation. Despite demonstrating good generalizability, OFT still uses a
fairly large number of trainable parameters due to the high dimensionality of
orthogonal matrices. To address this, we start by examining OFT from an
information transmission perspective, and then identify a few key desiderata
that enable better parameter-efficiency. Inspired by how the Cooley-Tukey fast
Fourier transform algorithm enables efficient information transmission, we
propose an efficient orthogonal parameterization using butterfly structures. We
apply this parameterization to OFT, creating a novel parameter-efficient
finetuning method, called Orthogonal Butterfly (BOFT). By subsuming OFT as a
special case, BOFT introduces a generalized orthogonal finetuning framework.
Finally, we conduct an extensive empirical study of adapting large vision
transformers, large language models, and text-to-image diffusion models to
various downstream tasks in vision and language.
JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal
  Language Models
Authors: name: Zihao Wang, name: Shaofei Cai, name: Anji Liu, name: Yonggang Jin, name: Jinbing Hou, name: Bowei Zhang, name: Haowei Lin, name: Zhaofeng He, name: Zilong Zheng, name: Yaodong Yang, name: Xiaojian Ma, name: Yitao Liang
Published: 2023-11-10 11:17:58

Achieving human-like planning and control with multimodal observations in an
open world is a key milestone for more functional generalist agents. Existing
approaches can handle certain long-horizon tasks in an open world. However,
they still struggle when the number of open-world tasks could potentially be
infinite and lack the capability to progressively enhance task completion as
game time progresses. We introduce JARVIS-1, an open-world agent that can
perceive multimodal input (visual observations and human instructions),
generate sophisticated plans, and perform embodied control, all within the
popular yet challenging open-world Minecraft universe. Specifically, we develop
JARVIS-1 on top of pre-trained multimodal language models, which map visual
observations and textual instructions to plans. The plans will be ultimately
dispatched to the goal-conditioned controllers. We outfit JARVIS-1 with a
multimodal memory, which facilitates planning using both pre-trained knowledge
and its actual game survival experiences. In our experiments, JARVIS-1 exhibits
nearly perfect performances across over 200 varying tasks from the Minecraft
Universe Benchmark, ranging from entry to intermediate levels. JARVIS-1 has
achieved a completion rate of 12.5% in the long-horizon diamond pickaxe task.
This represents a significant increase up to 5 times compared to previous
records. Furthermore, we show that JARVIS-1 is able to self-improve
following a life-long learning paradigm thanks to multimodal memory, sparking a
more general intelligence and improved autonomy. The project page is available
at https://craftjarvis-jarvis1.github.io.
ADaPT: As-Needed Decomposition and Planning with Language Models
Authors: name: Archiki Prasad, name: Alexander Koller, name: Mareike Hartmann, name: Peter Clark, name: Ashish Sabharwal, name: Mohit Bansal, name: Tushar Khot
Published: 2023-11-08 17:59:15

Large Language Models (LLMs) are increasingly being used for interactive
decision-making tasks requiring planning and adapting to the environment.
Recent works employ LLMs-as-agents in broadly two ways: iteratively determining
the next action (iterative executors) or generating plans and executing
sub-tasks using LLMs (plan-and-execute). However, these methods struggle with
task complexity, as the inability to execute any sub-task may lead to task
failure. To address these shortcomings, we introduce As-Needed Decomposition
and Planning for complex Tasks (ADaPT), an approach that explicitly plans and
decomposes complex sub-tasks as-needed, i.e., when the LLM is unable to execute
them. ADaPT recursively decomposes sub-tasks to adapt to both task complexity
and LLM capability. Our results demonstrate that ADaPT substantially
outperforms established strong baselines, achieving success rates up to 28.3%
higher in ALFWorld, 27% in WebShop, and 33% in TextCraft -- a novel
compositional dataset that we introduce. Through extensive analysis, we
illustrate the importance of multilevel decomposition and establish that ADaPT
dynamically adjusts to the capabilities of the executor LLM as well as to task
complexity.
Prompt Engineering a Prompt Engineer
Authors: name: Qinyuan Ye, name: Maxamed Axmed, name: Reid Pryzant, name: Fereshte Khani
Published: 2023-11-09 08:00:32

Prompt engineering is a challenging yet crucial task for optimizing the
performance of large language models (LLMs). It requires complex reasoning to
examine the model's errors, hypothesize what is missing or misleading in the
current prompt, and communicate the task with clarity. While recent works
indicate that LLMs can be meta-prompted to perform automatic prompt
engineering, their potentials may not be fully untapped due to the lack of
sufficient guidance to elicit complex reasoning capabilities in LLMs in the
meta-prompt. In this work, we investigate the problem of "prompt engineering a
prompt engineer" -- constructing a meta-prompt that more effectively guides
LLMs to perform automatic prompt engineering. We introduce and analyze key
components, such as a step-by-step reasoning template and context
specification, which lead to improved performance. In addition, inspired by
common optimization concepts such as batch size, step size and momentum, we
introduce their verbalized counterparts to the meta-prompt and investigate
their effects. Our final method, named PE2, finds a prompt that outperforms
"let's think step by step" by 6.3% on the MultiArith dataset and 3.1% on the
GSM8K dataset. To demonstrate its versatility, we apply PE2 to the Instruction
Induction benchmark, a suite of counterfactual tasks, and a lengthy, real-world
industrial prompt. In these settings, PE2 achieves strong performance and
outperforms prior automatic prompt engineering baselines. Further, we show that
PE2 makes meaningful and targeted prompt edits, amends erroneous or incomplete
prompts, and presents non-trivial counterfactual reasoning abilities.
FinGPT: Large Generative Models for a Small Language
Authors: name: Risto Luukkonen, name: Ville Komulainen, name: Jouni Luoma, name: Anni Eskelinen, name: Jenna Kanerva, name: Hanna-Mari Kupari, name: Filip Ginter, name: Veronika Laippala, name: Niklas Muennighoff, name: Aleksandra Piktus, name: Thomas Wang, name: Nouamane Tazi, name: Teven Le Scao, name: Thomas Wolf, name: Osma Suominen, name: Samuli Sairanen, name: Mikko Merioksa, name: Jyrki Heinonen, name: Aija Vahtola, name: Samuel Antao, name: Sampo Pyysalo
Published: 2023-11-03 08:05:04

Large language models (LLMs) excel in many tasks in NLP and beyond, but most
open models have very limited coverage of smaller languages and LLM work tends
to focus on languages where nearly unlimited data is available for pretraining.
In this work, we study the challenges of creating LLMs for Finnish, a language
spoken by less than 0.1% of the world population. We compile an extensive
dataset of Finnish combining web crawls, news, social media and eBooks. We
pursue two approaches to pretrain models: 1) we train seven monolingual models
from scratch (186M to 13B parameters) dubbed FinGPT, 2) we continue the
pretraining of the multilingual BLOOM model on a mix of its original training
data and Finnish, resulting in a 176 billion parameter model we call BLUUMI.
For model evaluation, we introduce FIN-bench, a version of BIG-bench with
Finnish tasks. We also assess other model qualities such as toxicity and bias.
Our models and tools are openly available at https://turkunlp.org/gpt3-finnish.