Orionnnn/hub-stats huggingface dataset:
Knowledge Mechanisms in Large Language Models: A Survey and Perspective
Authors: name: Mengru Wang, name: Yunzhi Yao, name: Ziwen Xu, name: Shuofei Qiao, name: Shumin Deng, name: Peng Wang, name: Xiang Chen, name: Jia-Chen Gu, name: Yong Jiang, name: Pengjun Xie, name: Fei Huang, name: Huajun Chen, name: Ningyu Zhang
Published: 2024-07-22 06:15:59

Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial
for advancing towards trustworthy AGI. This paper reviews knowledge mechanism
analysis from a novel taxonomy including knowledge utilization and evolution.
Knowledge utilization delves into the mechanism of memorization, comprehension
and application, and creation. Knowledge evolution focuses on the dynamic
progression of knowledge within individual and group LLMs. Moreover, we discuss
what knowledge LLMs have learned, the reasons for the fragility of parametric
knowledge, and the potential dark knowledge (hypothesis) that will be
challenging to address. We hope this work can help understand knowledge in LLMs
and provide insights for future research.
LongVideoBench: A Benchmark for Long-context Interleaved Video-Language
  Understanding
Authors: name: Haoning Wu, name: Dongxu Li, name: Bei Chen, name: Junnan Li
Published: 2024-07-22 16:00:55

Large multimodal models (LMMs) are processing increasingly longer and richer
inputs. Albeit the progress, few public benchmark is available to measure such
development. To mitigate this gap, we introduce LongVideoBench, a
question-answering benchmark that features video-language interleaved inputs up
to an hour long. Our benchmark includes 3,763 varying-length web-collected
videos with their subtitles across diverse themes, designed to comprehensively
evaluate LMMs on long-term multimodal understanding. To achieve this, we
interpret the primary challenge as to accurately retrieve and reason over
detailed multimodal information from long inputs. As such, we formulate a novel
video question-answering task termed referring reasoning. Specifically, as part
of the question, it contains a referring query that references related video
contexts, called referred context. The model is then required to reason over
relevant video details from the referred context. Following the paradigm of
referring reasoning, we curate 6,678 human-annotated multiple-choice questions
in 17 fine-grained categories, establishing one of the most comprehensive
benchmarks for long-form video understanding. Evaluations suggest that the
LongVideoBench presents significant challenges even for the most advanced
proprietary models (e.g. GPT-4o, Gemini-1.5-Pro, GPT-4-Turbo), while their
open-source counterparts show an even larger performance gap. In addition, our
results indicate that model performance on the benchmark improves only when
they are capable of processing more frames, positioning LongVideoBench as a
valuable benchmark for evaluating future-generation long-context LMMs.
Artist: Aesthetically Controllable Text-Driven Stylization without
  Training
Authors: name: Ruixiang Jiang, name: Changwen Chen
Published: 2024-07-22 17:58:05

Diffusion models entangle content and style generation during the denoising
process, leading to undesired content modification when directly applied to
stylization tasks. Existing methods struggle to effectively control the
diffusion model to meet the aesthetic-level requirements for stylization. In
this paper, we introduce Artist, a training-free approach that
aesthetically controls the content and style generation of a pretrained
diffusion model for text-driven stylization. Our key insight is to disentangle
the denoising of content and style into separate diffusion processes while
sharing information between them. We propose simple yet effective content and
style control methods that suppress style-irrelevant content generation,
resulting in harmonious stylization results. Extensive experiments demonstrate
that our method excels at achieving aesthetic-level stylization requirements,
preserving intricate details in the content image and aligning well with the
style prompt. Furthermore, we showcase the highly controllability of the
stylization strength from various perspectives. Code will be released, project
home page: https://DiffusionArtist.github.io
Cinemo: Consistent and Controllable Image Animation with Motion
  Diffusion Models
Authors: name: Xin Ma, name: Yaohui Wang, name: Gengyu Jia, name: Xinyuan Chen, name: Yuan-Fang Li, name: Cunjian Chen, name: Yu Qiao
Published: 2024-07-22 14:00:03

Diffusion models have achieved great progress in image animation due to
powerful generative capabilities. However, maintaining spatio-temporal
consistency with detailed information from the input static image over time
(e.g., style, background, and object of the input static image) and ensuring
smoothness in animated video narratives guided by textual prompts still remains
challenging. In this paper, we introduce Cinemo, a novel image animation
approach towards achieving better motion controllability, as well as stronger
temporal consistency and smoothness. In general, we propose three effective
strategies at the training and inference stages of Cinemo to accomplish our
goal. At the training stage, Cinemo focuses on learning the distribution of
motion residuals, rather than directly predicting subsequent via a motion
diffusion model. Additionally, a structural similarity index-based strategy is
proposed to enable Cinemo to have better controllability of motion intensity.
At the inference stage, a noise refinement technique based on discrete cosine
transformation is introduced to mitigate sudden motion changes. Such three
strategies enable Cinemo to produce highly consistent, smooth, and
motion-controllable results. Compared to previous methods, Cinemo offers
simpler and more precise user controllability. Extensive experiments against
several state-of-the-art methods, including both commercial tools and research
approaches, across multiple metrics, demonstrate the effectiveness and
superiority of our proposed approach.
Local All-Pair Correspondence for Point Tracking
Authors: name: Seokju Cho, name: Jiahui Huang, name: Jisu Nam, name: Honggyu An, name: Seungryong Kim, name: Joon-Young Lee
Published: 2024-07-22 06:49:56

We introduce LocoTrack, a highly accurate and efficient model designed for
the task of tracking any point (TAP) across video sequences. Previous
approaches in this task often rely on local 2D correlation maps to establish
correspondences from a point in the query image to a local region in the target
image, which often struggle with homogeneous regions or repetitive features,
leading to matching ambiguities. LocoTrack overcomes this challenge with a
novel approach that utilizes all-pair correspondences across regions, i.e.,
local 4D correlation, to establish precise correspondences, with bidirectional
correspondence and matching smoothness significantly enhancing robustness
against ambiguities. We also incorporate a lightweight correlation encoder to
enhance computational efficiency, and a compact Transformer architecture to
integrate long-term temporal information. LocoTrack achieves unmatched accuracy
on all TAP-Vid benchmarks and operates at a speed almost 6 times faster than
the current state-of-the-art.
ThermalNeRF: Thermal Radiance Fields
Authors: name: Yvette Y. Lin, name: Xin-Yi Pan, name: Sara Fridovich-Keil, name: Gordon Wetzstein
Published: 2024-07-22 02:51:29

Thermal imaging has a variety of applications, from agricultural monitoring
to building inspection to imaging under poor visibility, such as in low light,
fog, and rain. However, reconstructing thermal scenes in 3D presents several
challenges due to the comparatively lower resolution and limited features
present in long-wave infrared (LWIR) images. To overcome these challenges, we
propose a unified framework for scene reconstruction from a set of LWIR and RGB
images, using a multispectral radiance field to represent a scene viewed by
both visible and infrared cameras, thus leveraging information across both
spectra. We calibrate the RGB and infrared cameras with respect to each other,
as a preprocessing step using a simple calibration target. We demonstrate our
method on real-world sets of RGB and LWIR photographs captured from a handheld
thermal camera, showing the effectiveness of our method at scene representation
across the visible and infrared spectra. We show that our method is capable of
thermal super-resolution, as well as visually removing obstacles to reveal
objects that are occluded in either the RGB or thermal channels. Please see
https://yvette256.github.io/thermalnerf for video results as well as our code
and dataset release.
HoloDreamer: Holistic 3D Panoramic World Generation from Text
  Descriptions
Authors: name: Haiyang Zhou, name: Xinhua Cheng, name: Wangbo Yu, name: Yonghong Tian, name: Li Yuan
Published: 2024-07-21 14:52:51

3D scene generation is in high demand across various domains, including
virtual reality, gaming, and the film industry. Owing to the powerful
generative capabilities of text-to-image diffusion models that provide reliable
priors, the creation of 3D scenes using only text prompts has become viable,
thereby significantly advancing researches in text-driven 3D scene generation.
In order to obtain multiple-view supervision from 2D diffusion models,
prevailing methods typically employ the diffusion model to generate an initial
local image, followed by iteratively outpainting the local image using
diffusion models to gradually generate scenes. Nevertheless, these
outpainting-based approaches prone to produce global inconsistent scene
generation results without high degree of completeness, restricting their
broader applications. To tackle these problems, we introduce HoloDreamer, a
framework that first generates high-definition panorama as a holistic
initialization of the full 3D scene, then leverage 3D Gaussian Splatting
(3D-GS) to quickly reconstruct the 3D scene, thereby facilitating the creation
of view-consistent and fully enclosed 3D scenes. Specifically, we propose
Stylized Equirectangular Panorama Generation, a pipeline that combines multiple
diffusion models to enable stylized and detailed equirectangular panorama
generation from complex text prompts. Subsequently, Enhanced Two-Stage Panorama
Reconstruction is introduced, conducting a two-stage optimization of 3D-GS to
inpaint the missing region and enhance the integrity of the scene.
Comprehensive experiments demonstrated that our method outperforms prior works
in terms of overall visual consistency and harmony as well as reconstruction
quality and rendering robustness when generating fully enclosed scenes.
MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
  Generation
Authors: name: Yun-Han Lan, name: Wen-Yi Hsiao, name: Hao-Chung Cheng, name: Yi-Hsuan Yang
Published: 2024-07-21 05:27:53

Existing text-to-music models can produce high-quality audio with great
diversity. However, textual prompts alone cannot precisely control temporal
musical features such as chords and rhythm of the generated music. To address
this challenge, we introduce MusiConGen, a temporally-conditioned
Transformer-based text-to-music model that builds upon the pretrained MusicGen
framework. Our innovation lies in an efficient finetuning mechanism, tailored
for consumer-grade GPUs, that integrates automatically-extracted rhythm and
chords as the condition signal. During inference, the condition can either be
musical features extracted from a reference audio signal, or be user-defined
symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
on two datasets -- one derived from extracted features and the other from
user-created inputs -- demonstrates that MusiConGen can generate realistic
backing track music that aligns well with the specified conditions. We
open-source the code and model checkpoints, and provide audio examples online,
https://musicongen.github.io/musicongen_demo/.
VideoGameBunny: Towards vision assistants for video games
Authors: name: Mohammad Reza Taesiri, name: Cor-Paul Bezemer
Published: 2024-07-21 23:31:57

Large multimodal models (LMMs) hold substantial promise across various
domains, from personal assistance in daily tasks to sophisticated applications
like medical diagnostics. However, their capabilities have limitations in the
video game domain, such as challenges with scene understanding, hallucinations,
and inaccurate descriptions of video game content, especially in open-source
models. This paper describes the development of VideoGameBunny, a LLaVA-style
model based on Bunny, specifically tailored for understanding images from video
games. We release intermediate checkpoints, training logs, and an extensive
dataset comprising 185,259 video game images from 413 titles, along with
389,565 image-instruction pairs that include image captions, question-answer
pairs, and a JSON representation of 16 elements of 136,974 images. Our
experiments show that our high quality game-related data has the potential to
make a relatively small model outperform the much larger state-of-the-art model
LLaVa-1.6-34b (which has more than 4x the number of parameters). Our study
paves the way for future research in video game understanding on tasks such as
playing, commentary, and debugging. Code and data are available at
https://videogamebunny.github.io/
Conditioned Language Policy: A General Framework for Steerable
  Multi-Objective Finetuning
Authors: name: Kaiwen Wang, name: Rahul Kidambi, name: Ryan Sullivan, name: Alekh Agarwal, name: Christoph Dann, name: Andrea Michi, name: Marco Gelmi, name: Yunxuan Li, name: Raghav Gupta, name: Avinava Dubey, name: Alexandre Ramé, name: Johan Ferret, name: Geoffrey Cideron, name: Le Hou, name: Hongkun Yu, name: Amr Ahmed, name: Aranyak Mehta, name: Léonard Hussenot, name: Olivier Bachem, name: Edouard Leurent
Published: 2024-07-22 16:13:38

Reward-based finetuning is crucial for aligning language policies with
intended behaviors (e.g., creativity and safety). A key challenge here is to
develop steerable language models that trade-off multiple (conflicting)
objectives in a flexible and efficient manner. This paper presents Conditioned
Language Policy (CLP), a general framework for finetuning language models on
multiple objectives. Building on techniques from multi-task training and
parameter-efficient finetuning, CLP can learn steerable models that effectively
trade-off conflicting objectives at inference time. Notably, this does not
require training or maintaining multiple models to achieve different trade-offs
between the objectives. Through an extensive set of experiments and ablations,
we show that the CLP framework learns steerable models that outperform and
Pareto-dominate the current state-of-the-art approaches for multi-objective
finetuning.
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language
  Models
Authors: name: Mingze Xu, name: Mingfei Gao, name: Zhe Gan, name: Hong-You Chen, name: Zhengfeng Lai, name: Haiming Gang, name: Kai Kang, name: Afshin Dehghan
Published: 2024-07-22 17:58:04

We propose SlowFast-LLaVA (or SF-LLaVA for short), a training-free video
large language model (LLM) that can jointly capture the detailed spatial
semantics and long-range temporal context without exceeding the token budget of
commonly used LLMs. This is realized by using a two-stream SlowFast design of
inputs for Video LLMs to aggregate features from sampled video frames in an
effective way. Specifically, the Slow pathway extracts features at a low frame
rate while keeping as many spatial details as possible (e.g., with 24x24
tokens), and the Fast pathway operates on a high frame rate but uses a larger
spatial pooling stride (e.g., downsampling 6x) to focus on the motion cues. As
a result, this design allows us to adequately capture both spatial and temporal
features that are beneficial for understanding details along the video.
Experimental results show that SF-LLaVA outperforms existing training-free
methods on a wide range of video tasks. On some benchmarks, it achieves
comparable or even better performance compared to state-of-the-art Video LLMs
that are fine-tuned on video datasets.
GET-Zero: Graph Embodiment Transformer for Zero-shot Embodiment
  Generalization
Authors: name: Austin Patel, name: Shuran Song
Published: 2024-07-20 22:39:12

This paper introduces GET-Zero, a model architecture and training procedure
for learning an embodiment-aware control policy that can immediately adapt to
new hardware changes without retraining. To do so, we present Graph Embodiment
Transformer (GET), a transformer model that leverages the embodiment graph
connectivity as a learned structural bias in the attention mechanism. We use
behavior cloning to distill demonstration data from embodiment-specific expert
policies into an embodiment-aware GET model that conditions on the hardware
configuration of the robot to make control decisions. We conduct a case study
on a dexterous in-hand object rotation task using different configurations of a
four-fingered robot hand with joints removed and with link length extensions.
Using the GET model along with a self-modeling loss enables GET-Zero to
zero-shot generalize to unseen variation in graph structure and link length,
yielding a 20% improvement over baseline methods. All code and qualitative
video results are on https://get-zero-paper.github.io
Temporal Residual Jacobians For Rig-free Motion Transfer
Authors: name: Sanjeev Muralikrishnan, name: Niladri Shekhar Dutt, name: Siddhartha Chaudhuri, name: Noam Aigerman, name: Vladimir Kim, name: Matthew Fisher, name: Niloy J. Mitra
Published: 2024-07-20 18:29:22

We introduce Temporal Residual Jacobians as a novel representation to enable
data-driven motion transfer. Our approach does not assume access to any rigging
or intermediate shape keyframes, produces geometrically and temporally
consistent motions, and can be used to transfer long motion sequences. Central
to our approach are two coupled neural networks that individually predict local
geometric and temporal changes that are subsequently integrated, spatially and
temporally, to produce the final animated meshes. The two networks are jointly
trained, complement each other in producing spatial and temporal signals, and
are supervised directly with 3D positional information. During inference, in
the absence of keyframes, our method essentially solves a motion extrapolation
problem. We test our setup on diverse meshes (synthetic and scanned shapes) to
demonstrate its superiority in generating realistic and natural-looking
animations on unseen body shapes against SoTA alternatives. Supplemental video
and code are available at https://temporaljacobians.github.io/ .
NNsight and NDIF: Democratizing Access to Foundation Model Internals
Authors: name: Jaden Fiotto-Kaufman, name: Alexander R Loftus, name: Eric Todd, name: Jannik Brinkmann, name: Caden Juang, name: Koyena Pal, name: Can Rager, name: Aaron Mueller, name: Samuel Marks, name: Arnab Sen Sharma, name: Francesca Lucchetti, name: Michael Ripa, name: Adam Belfki, name: Nikhil Prakash, name: Sumeet Multani, name: Carla Brodley, name: Arjun Guha, name: Jonathan Bell, name: Byron Wallace, name: David Bau
Published: 2024-07-18 17:59:01

The enormous scale of state-of-the-art foundation models has limited their
accessibility to scientists, because customized experiments at large model
sizes require costly hardware and complex engineering that is impractical for
most researchers. To alleviate these problems, we introduce NNsight, an
open-source Python package with a simple, flexible API that can express
interventions on any PyTorch model by building computation graphs. We also
introduce NDIF, a collaborative research platform providing researchers access
to foundation-scale LLMs via the NNsight API. Code, documentation, and
tutorials are available at https://www.nnsight.net.
Internal Consistency and Self-Feedback in Large Language Models: A
  Survey
Authors: name: Xun Liang, name: Shichao Song, name: Zifan Zheng, name: Hanyu Wang, name: Qingchen Yu, name: Xunkai Li, name: Rong-Hua Li, name: Feiyu Xiong, name: Zhiyu Li
Published: 2024-07-19 17:59:03

Large language models (LLMs) are expected to respond accurately but often
exhibit deficient reasoning or generate hallucinatory content. To address
these, studies prefixed with ``Self-'' such as Self-Consistency, Self-Improve,
and Self-Refine have been initiated. They share a commonality: involving LLMs
evaluating and updating itself to mitigate the issues. Nonetheless, these
efforts lack a unified perspective on summarization, as existing surveys
predominantly focus on categorization without examining the motivations behind
these works.
  In this paper, we summarize a theoretical framework, termed Internal
Consistency, which offers unified explanations for phenomena such as the lack
of reasoning and the presence of hallucinations. Internal Consistency assesses
the coherence among LLMs' latent layer, decoding layer, and response layer
based on sampling methodologies. Expanding upon the Internal Consistency
framework, we introduce a streamlined yet effective theoretical framework
capable of mining Internal Consistency, named Self-Feedback. The Self-Feedback
framework consists of two modules: Self-Evaluation and Self-Update. This
framework has been employed in numerous studies.
  We systematically classify these studies by tasks and lines of work;
summarize relevant evaluation methods and benchmarks; and delve into the
concern, ``Does Self-Feedback Really Work?'' We propose several critical
viewpoints, including the ``Hourglass Evolution of Internal Consistency'',
``Consistency Is (Almost) Correctness'' hypothesis, and ``The Paradox of Latent
and Explicit Reasoning''. Furthermore, we outline promising directions for
future research. We have open-sourced the experimental code, reference list,
and statistical data, available at
https://github.com/IAAR-Shanghai/ICSFSurvey.
Qalam : A Multimodal LLM for Arabic Optical Character and Handwriting
  Recognition
Authors: name: Gagan Bhatia, name: El Moatez Billah Nagoudi, name: Fakhraddin Alwajih, name: Muhammad Abdul-Mageed
Published: 2024-07-18 14:31:09

Arabic Optical Character Recognition (OCR) and Handwriting Recognition (HWR)
pose unique challenges due to the cursive and context-sensitive nature of the
Arabic script. This study introduces Qalam, a novel foundation model designed
for Arabic OCR and HWR, built on a SwinV2 encoder and RoBERTa decoder
architecture. Our model significantly outperforms existing methods, achieving a
Word Error Rate (WER) of just 0.80% in HWR tasks and 1.18% in OCR tasks. We
train Qalam on a diverse dataset, including over 4.5 million images from Arabic
manuscripts and a synthetic dataset comprising 60k image-text pairs. Notably,
Qalam demonstrates exceptional handling of Arabic diacritics, a critical
feature in Arabic scripts. Furthermore, it shows a remarkable ability to
process high-resolution inputs, addressing a common limitation in current OCR
systems. These advancements underscore Qalam's potential as a leading solution
for Arabic script recognition, offering a significant leap in accuracy and
efficiency.