Orionnnn/hub-stats huggingface dataset:
Amuro & Char: Analyzing the Relationship between Pre-Training and
  Fine-Tuning of Large Language Models
Authors: name: Kaiser Sun, name: Mark Dredze
Published: 2024-08-13 06:28:43

The development of large language models leads to the formation of a
pre-train-then-align paradigm, in which the model is typically pre-trained on a
large text corpus and undergoes a tuning stage to align the model with human
preference or downstream tasks. In this work, we investigate the relationship
between pre-training and fine-tuning by fine-tuning multiple intermediate
pre-trained model checkpoints. Our results on 18 datasets suggest that i)
continual pre-training improves the model in a latent way that unveils after
fine-tuning; ii) with extra fine-tuning, the datasets that the model does not
demonstrate capability gain much more than those that the model performs well
during the pre-training stage; iii) although model benefits significantly
through supervised fine-tuning, it may forget previously known domain knowledge
and the tasks that are not seen during fine-tuning; iv) the model resembles
high sensitivity to evaluation prompts after supervised fine-tuning, but this
sensitivity can be alleviated by more pre-training.
MovieSum: An Abstractive Summarization Dataset for Movie Screenplays
Authors: name: Rohit Saxena, name: Frank Keller
Published: 2024-08-12 16:43:09

Movie screenplay summarization is challenging, as it requires an
understanding of long input contexts and various elements unique to movies.
Large language models have shown significant advancements in document
summarization, but they often struggle with processing long input contexts.
Furthermore, while television transcripts have received attention in recent
studies, movie screenplay summarization remains underexplored. To stimulate
research in this area, we present a new dataset, MovieSum, for abstractive
summarization of movie screenplays. This dataset comprises 2200 movie
screenplays accompanied by their Wikipedia plot summaries. We manually
formatted the movie screenplays to represent their structural elements.
Compared to existing datasets, MovieSum possesses several distinctive features:
(1) It includes movie screenplays, which are longer than scripts of TV
episodes. (2) It is twice the size of previous movie screenplay datasets. (3)
It provides metadata with IMDb IDs to facilitate access to additional external
knowledge. We also show the results of recently released large language models
applied to summarization on our dataset to provide a detailed baseline.
Layerwise Recurrent Router for Mixture-of-Experts
Authors: name: Zihan Qiu, name: Zeyu Huang, name: Shuang Cheng, name: Yizhi Zhou, name: Zili Wang, name: Ivan Titov, name: Jie Fu
Published: 2024-08-13 10:25:13

The scaling of large language models (LLMs) has revolutionized their
capabilities in various tasks, yet this growth must be matched with efficient
computational strategies. The Mixture-of-Experts (MoE) architecture stands out
for its ability to scale model size without significantly increasing training
costs. Despite their advantages, current MoE models often display parameter
inefficiency. For instance, a pre-trained MoE-based LLM with 52 billion
parameters might perform comparably to a standard model with 6.7 billion
parameters. Being a crucial part of MoE, current routers in different layers
independently assign tokens without leveraging historical routing information,
potentially leading to suboptimal token-expert combinations and the parameter
inefficiency problem. To alleviate this issue, we introduce the Layerwise
Recurrent Router for Mixture-of-Experts (RMoE). RMoE leverages a Gated
Recurrent Unit (GRU) to establish dependencies between routing decisions across
consecutive layers. Such layerwise recurrence can be efficiently parallelly
computed for input tokens and introduces negotiable costs. Our extensive
empirical evaluations demonstrate that RMoE-based language models consistently
outperform a spectrum of baseline models. Furthermore, RMoE integrates a novel
computation stage orthogonal to existing methods, allowing seamless
compatibility with other MoE architectures. Our analyses attribute RMoE's gains
to its effective cross-layer information sharing, which also improves expert
selection and diversity. Our code is at https://github.com/qiuzh20/RMoE
SlotLifter: Slot-guided Feature Lifting for Learning Object-centric
  Radiance Fields
Authors: name: Yu Liu, name: Baoxiong Jia, name: Yixin Chen, name: Siyuan Huang
Published: 2024-08-13 07:51:37

The ability to distill object-centric abstractions from intricate visual
scenes underpins human-level generalization. Despite the significant progress
in object-centric learning methods, learning object-centric representations in
the 3D physical world remains a crucial challenge. In this work, we propose
SlotLifter, a novel object-centric radiance model addressing scene
reconstruction and decomposition jointly via slot-guided feature lifting. Such
a design unites object-centric learning representations and image-based
rendering methods, offering state-of-the-art performance in scene decomposition
and novel-view synthesis on four challenging synthetic and four complex
real-world datasets, outperforming existing 3D object-centric learning methods
by a large margin. Through extensive ablative studies, we showcase the efficacy
of designs in SlotLifter, revealing key insights for potential future
directions.
TacSL: A Library for Visuotactile Sensor Simulation and Learning
Authors: name: Iretiayo Akinola, name: Jie Xu, name: Jan Carius, name: Dieter Fox, name: Yashraj Narang
Published: 2024-08-12 21:44:43

For both humans and robots, the sense of touch, known as tactile sensing, is
critical for performing contact-rich manipulation tasks. Three key challenges
in robotic tactile sensing are 1) interpreting sensor signals, 2) generating
sensor signals in novel scenarios, and 3) learning sensor-based policies. For
visuotactile sensors, interpretation has been facilitated by their close
relationship with vision sensors (e.g., RGB cameras). However, generation is
still difficult, as visuotactile sensors typically involve contact,
deformation, illumination, and imaging, all of which are expensive to simulate;
in turn, policy learning has been challenging, as simulation cannot be
leveraged for large-scale data collection. We present TacSL
(taxel), a library for GPU-based visuotactile sensor simulation and
learning. TacSL can be used to simulate visuotactile images and
extract contact-force distributions over 200times faster than the prior
state-of-the-art, all within the widely-used Isaac Gym simulator. Furthermore,
TacSL provides a learning toolkit containing multiple sensor models,
contact-intensive training environments, and online/offline algorithms that can
facilitate policy learning for sim-to-real applications. On the algorithmic
side, we introduce a novel online reinforcement-learning algorithm called
asymmetric actor-critic distillation (\sysName), designed to effectively and
efficiently learn tactile-based policies in simulation that can transfer to the
real world. Finally, we demonstrate the utility of our library and algorithms
by evaluating the benefits of distillation and multimodal sensing for
contact-rich manip ulation tasks, and most critically, performing sim-to-real
transfer. Supplementary videos and results are at
https://iakinola23.github.io/tacsl/.
UniT: Unified Tactile Representation for Robot Learning
Authors: name: Zhengtong Xu, name: Raghava Uppuluri, name: Xinwei Zhang, name: Cael Fitch, name: Philip Glen Crandall, name: Wan Shou, name: Dongyi Wang, name: Yu She
Published: 2024-08-12 20:29:09

UniT is a novel approach to tactile representation learning, using VQVAE to
learn a compact latent space and serve as the tactile representation. It uses
tactile images obtained from a single simple object to train the representation
with transferability and generalizability. This tactile representation can be
zero-shot transferred to various downstream tasks, including perception tasks
and manipulation policy learning. Our benchmarking on an in-hand 3D pose
estimation task shows that UniT outperforms existing visual and tactile
representation learning methods. Additionally, UniT's effectiveness in policy
learning is demonstrated across three real-world tasks involving diverse
manipulated objects and complex robot-object-environment interactions. Through
extensive experimentation, UniT is shown to be a simple-to-train,
plug-and-play, yet widely effective method for tactile representation learning.
For more details, please refer to our open-source repository
https://github.com/ZhengtongXu/UniT and the project website
https://zhengtongxu.github.io/unifiedtactile.github.io/.
DC3DO: Diffusion Classifier for 3D Objects
Authors: name: Nursena Koprucu, name: Meher Shashwat Nigam, name: Shicheng Xu, name: Biruk Abere, name: Gabriele Dominici, name: Andrew Rodriguez, name: Sharvaree Vadgam, name: Berfin Inal, name: Alberto Tono
Published: 2024-08-13 07:35:56

Inspired by Geoffrey Hinton emphasis on generative modeling, To recognize
shapes, first learn to generate them, we explore the use of 3D diffusion models
for object classification. Leveraging the density estimates from these models,
our approach, the Diffusion Classifier for 3D Objects (DC3DO), enables
zero-shot classification of 3D shapes without additional training. On average,
our method achieves a 12.5 percent improvement compared to its multiview
counterparts, demonstrating superior multimodal reasoning over discriminative
approaches. DC3DO employs a class-conditional diffusion model trained on
ShapeNet, and we run inferences on point clouds of chairs and cars. This work
highlights the potential of generative models in 3D object classification.
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
Authors: name: Yushi Bai, name: Jiajie Zhang, name: Xin Lv, name: Linzhi Zheng, name: Siqi Zhu, name: Lei Hou, name: Yuxiao Dong, name: Jie Tang, name: Juanzi Li
Published: 2024-08-13 17:46:12

Current long context large language models (LLMs) can process inputs up to
100,000 tokens, yet struggle to generate outputs exceeding even a modest length
of 2,000 words. Through controlled experiments, we find that the model's
effective generation length is inherently bounded by the sample it has seen
during supervised fine-tuning (SFT). In other words, their output limitation is
due to the scarcity of long-output examples in existing SFT datasets. To
address this, we introduce AgentWrite, an agent-based pipeline that decomposes
ultra-long generation tasks into subtasks, enabling off-the-shelf LLMs to
generate coherent outputs exceeding 20,000 words. Leveraging AgentWrite, we
construct LongWriter-6k, a dataset containing 6,000 SFT data with output
lengths ranging from 2k to 32k words. By incorporating this dataset into model
training, we successfully scale the output length of existing models to over
10,000 words while maintaining output quality. We also develop LongBench-Write,
a comprehensive benchmark for evaluating ultra-long generation capabilities.
Our 9B parameter model, further improved through DPO, achieves state-of-the-art
performance on this benchmark, surpassing even much larger proprietary models.
In general, our work demonstrates that existing long context LLM already
possesses the potential for a larger output window--all you need is data with
extended output during model alignment to unlock this capability. Our code &
models are at: https://github.com/THUDM/LongWriter.
OpenResearcher: Unleashing AI for Accelerated Scientific Research
Authors: name: Yuxiang Zheng, name: Shichao Sun, name: Lin Qiu, name: Dongyu Ru, name: Cheng Jiayang, name: Xuefeng Li, name: Jifan Lin, name: Binjie Wang, name: Yun Luo, name: Renjie Pan, name: Yang Xu, name: Qingkai Min, name: Zizhao Zhang, name: Yiwen Wang, name: Wenjie Li, name: Pengfei Liu
Published: 2024-08-13 14:59:44

The rapid growth of scientific literature imposes significant challenges for
researchers endeavoring to stay updated with the latest advancements in their
fields and delve into new areas. We introduce OpenResearcher, an innovative
platform that leverages Artificial Intelligence (AI) techniques to accelerate
the research process by answering diverse questions from researchers.
OpenResearcher is built based on Retrieval-Augmented Generation (RAG) to
integrate Large Language Models (LLMs) with up-to-date, domain-specific
knowledge. Moreover, we develop various tools for OpenResearcher to understand
researchers' queries, search from the scientific literature, filter retrieved
information, provide accurate and comprehensive answers, and self-refine these
answers. OpenResearcher can flexibly use these tools to balance efficiency and
effectiveness. As a result, OpenResearcher enables researchers to save time and
increase their potential to discover new insights and drive scientific
breakthroughs. Demo, video, and code are available at:
https://github.com/GAIR-NLP/OpenResearcher.
Your Context Is Not an Array: Unveiling Random Access Limitations in
  Transformers
Authors: name: MohammadReza Ebrahimi, name: Sunny Panchal, name: Roland Memisevic
Published: 2024-08-10 10:12:09

Despite their recent successes, Transformer-based large language models show
surprising failure modes. A well-known example of such failure modes is their
inability to length-generalize: solving problem instances at inference time
that are longer than those seen during training. In this work, we further
explore the root cause of this failure by performing a detailed analysis of
model behaviors on the simple parity task. Our analysis suggests that length
generalization failures are intricately related to a model's inability to
perform random memory accesses within its context window. We present supporting
evidence for this hypothesis by demonstrating the effectiveness of
methodologies that circumvent the need for indexing or that enable random token
access indirectly, through content-based addressing. We further show where and
how the failure to perform random memory access manifests through attention map
visualizations.
VisualAgentBench: Towards Large Multimodal Models as Visual Foundation
  Agents
Authors: name: Xiao Liu, name: Tianjie Zhang, name: Yu Gu, name: Iat Long Iong, name: Yifan Xu, name: Xixuan Song, name: Shudan Zhang, name: Hanyu Lai, name: Xinyi Liu, name: Hanlin Zhao, name: Jiadai Sun, name: Xinyue Yang, name: Yu Yang, name: Zehan Qi, name: Shuntian Yao, name: Xueqiao Sun, name: Siyi Cheng, name: Qinkai Zheng, name: Hao Yu, name: Hanchen Zhang, name: Wenyi Hong, name: Ming Ding, name: Lihang Pan, name: Xiaotao Gu, name: Aohan Zeng, name: Zhengxiao Du, name: Chan Hee Song, name: Yu Su, name: Yuxiao Dong, name: Jie Tang
Published: 2024-08-12 17:44:17

Large Multimodal Models (LMMs) have ushered in a new era in artificial
intelligence, merging capabilities in both language and vision to form highly
capable Visual Foundation Agents. These agents are postulated to excel across a
myriad of tasks, potentially approaching general artificial intelligence.
However, existing benchmarks fail to sufficiently challenge or showcase the
full potential of LMMs in complex, real-world environments. To address this
gap, we introduce VisualAgentBench (VAB), a comprehensive and pioneering
benchmark specifically designed to train and evaluate LMMs as visual foundation
agents across diverse scenarios, including Embodied, Graphical User Interface,
and Visual Design, with tasks formulated to probe the depth of LMMs'
understanding and interaction capabilities. Through rigorous testing across
nine proprietary LMM APIs and eight open models, we demonstrate the
considerable yet still developing agent capabilities of these models.
Additionally, VAB constructs a trajectory training set constructed through
hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and
Human Demonstrations, promoting substantial performance improvements in LMMs
through behavior cloning. Our work not only aims to benchmark existing models
but also provides a solid foundation for future development into visual
foundation agents. Code, train \& test data, and part of fine-tuned open LMMs
are available at https://github.com/THUDM/VisualAgentBench.
Body Transformer: Leveraging Robot Embodiment for Policy Learning
Authors: name: Carmelo Sferrazza, name: Dun-Ming Huang, name: Fangchen Liu, name: Jongmin Lee, name: Pieter Abbeel
Published: 2024-08-12 17:31:28

In recent years, the transformer architecture has become the de facto
standard for machine learning algorithms applied to natural language processing
and computer vision. Despite notable evidence of successful deployment of this
architecture in the context of robot learning, we claim that vanilla
transformers do not fully exploit the structure of the robot learning problem.
Therefore, we propose Body Transformer (BoT), an architecture that leverages
the robot embodiment by providing an inductive bias that guides the learning
process. We represent the robot body as a graph of sensors and actuators, and
rely on masked attention to pool information throughout the architecture. The
resulting architecture outperforms the vanilla transformer, as well as the
classical multilayer perceptron, in terms of task completion, scaling
properties, and computational efficiency when representing either imitation or
reinforcement learning policies. Additional material including the open-source
code is available at https://sferrazza.cc/bot_site.
Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers
Authors: name: Zhenting Qi, name: Mingyuan Ma, name: Jiahang Xu, name: Li Lyna Zhang, name: Fan Yang, name: Mao Yang
Published: 2024-08-12 14:42:13

This paper introduces rStar, a self-play mutual reasoning approach that
significantly improves reasoning capabilities of small language models (SLMs)
without fine-tuning or superior models. rStar decouples reasoning into a
self-play mutual generation-discrimination process. First, a target SLM
augments the Monte Carlo Tree Search (MCTS) with a rich set of human-like
reasoning actions to construct higher quality reasoning trajectories. Next,
another SLM, with capabilities similar to the target SLM, acts as a
discriminator to verify each trajectory generated by the target SLM. The
mutually agreed reasoning trajectories are considered mutual consistent, thus
are more likely to be correct. Extensive experiments across five SLMs
demonstrate rStar can effectively solve diverse reasoning problems, including
GSM8K, GSM-Hard, MATH, SVAMP, and StrategyQA. Remarkably, rStar boosts GSM8K
accuracy from 12.51% to 63.91% for LLaMA2-7B, from 36.46% to 81.88% for
Mistral-7B, from 74.53% to 91.13% for LLaMA3-8B-Instruct. Code will be
available at https://github.com/zhentingqi/rStar.
FruitNeRF: A Unified Neural Radiance Field based Fruit Counting
  Framework
Authors: name: Lukas Meyer, name: Andreas Gilson, name: Ute Schmidt, name: Marc Stamminger
Published: 2024-08-12 14:40:38

We introduce FruitNeRF, a unified novel fruit counting framework that
leverages state-of-the-art view synthesis methods to count any fruit type
directly in 3D. Our framework takes an unordered set of posed images captured
by a monocular camera and segments fruit in each image. To make our system
independent of the fruit type, we employ a foundation model that generates
binary segmentation masks for any fruit. Utilizing both modalities, RGB and
semantic, we train a semantic neural radiance field. Through uniform volume
sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By
applying cascaded clustering on the extracted point cloud, our approach
achieves precise fruit count.The use of neural radiance fields provides
significant advantages over conventional methods such as object tracking or
optical flow, as the counting itself is lifted into 3D. Our method prevents
double counting fruit and avoids counting irrelevant fruit.We evaluate our
methodology using both real-world and synthetic datasets. The real-world
dataset consists of three apple trees with manually counted ground truths, a
benchmark apple dataset with one row and ground truth fruit location, while the
synthetic dataset comprises various fruit types including apple, plum, lemon,
pear, peach, and mango.Additionally, we assess the performance of fruit
counting using the foundation model compared to a U-Net.
Med42-v2: A Suite of Clinical LLMs
Authors: name: Clément Christophe, name: Praveen K Kanithi, name: Tathagata Raha, name: Shadab Khan, name: Marco AF Pimentel
Published: 2024-08-12 13:37:31

Med42-v2 introduces a suite of clinical large language models (LLMs) designed
to address the limitations of generic models in healthcare settings. These
models are built on Llama3 architecture and fine-tuned using specialized
clinical data. They underwent multi-stage preference alignment to effectively
respond to natural prompts. While generic models are often preference-aligned
to avoid answering clinical queries as a precaution, Med42-v2 is specifically
trained to overcome this limitation, enabling its use in clinical settings.
Med42-v2 models demonstrate superior performance compared to the original
Llama3 models in both 8B and 70B parameter configurations and GPT-4 across
various medical benchmarks. These LLMs are developed to understand clinical
queries, perform reasoning tasks, and provide valuable assistance in clinical
environments. The models are now publicly available at
https://huggingface.co/m42-health{https://huggingface.co/m42-health}.
CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Authors: name: Zhuoyi Yang, name: Jiayan Teng, name: Wendi Zheng, name: Ming Ding, name: Shiyu Huang, name: Jiazheng Xu, name: Yuanming Yang, name: Wenyi Hong, name: Xiaohan Zhang, name: Guanyu Feng, name: Da Yin, name: Xiaotao Gu, name: Yuxuan Zhang, name: Weihan Wang, name: Yean Cheng, name: Ting Liu, name: Bin Xu, name: Yuxiao Dong, name: Jie Tang
Published: 2024-08-12 11:47:11

We introduce CogVideoX, a large-scale diffusion transformer model designed
for generating videos based on text prompts. To efficently model video data, we
propose to levearge a 3D Variational Autoencoder (VAE) to compress videos along
both spatial and temporal dimensions. To improve the text-video alignment, we
propose an expert transformer with the expert adaptive LayerNorm to facilitate
the deep fusion between the two modalities. By employing a progressive training
technique, CogVideoX is adept at producing coherent, long-duration videos
characterized by significant motions. In addition, we develop an effective
text-video data processing pipeline that includes various data preprocessing
strategies and a video captioning method. It significantly helps enhance the
performance of CogVideoX, improving both generation quality and semantic
alignment. Results show that CogVideoX demonstrates state-of-the-art
performance across both multiple machine metrics and human evaluations. The
model weights of both the 3D Causal VAE and CogVideoX are publicly available at
https://github.com/THUDM/CogVideo.