Orionnnn/hub-stats huggingface dataset:
NavGPT-2: Unleashing Navigational Reasoning Capability for Large
  Vision-Language Models
Authors: name: Gengze Zhou, name: Yicong Hong, name: Zun Wang, name: Xin Eric Wang, name: Qi Wu
Published: 2024-07-17 07:44:26

Capitalizing on the remarkable advancements in Large Language Models (LLMs),
there is a burgeoning initiative to harness LLMs for instruction following
robotic navigation. Such a trend underscores the potential of LLMs to
generalize navigational reasoning and diverse language understanding. However,
a significant discrepancy in agent performance is observed when integrating
LLMs in the Vision-and-Language navigation (VLN) tasks compared to previous
downstream specialist models. Furthermore, the inherent capacity of language to
interpret and facilitate communication in agent interactions is often
underutilized in these integrations. In this work, we strive to bridge the
divide between VLN-specialized models and LLM-based navigation paradigms, while
maintaining the interpretative prowess of LLMs in generating linguistic
navigational reasoning. By aligning visual content in a frozen LLM, we
encompass visual observation comprehension for LLMs and exploit a way to
incorporate LLMs and navigation policy networks for effective action
predictions and navigational reasoning. We demonstrate the data efficiency of
the proposed methods and eliminate the gap between LM-based agents and
state-of-the-art VLN specialists.
ThinkGrasp: A Vision-Language System for Strategic Part Grasping in
  Clutter
Authors: name: Yaoyao Qian, name: Xupeng Zhu, name: Ondrej Biza, name: Shuo Jiang, name: Linfeng Zhao, name: Haojie Huang, name: Yu Qi, name: Robert Platt
Published: 2024-07-16 01:06:33

Robotic grasping in cluttered environments remains a significant challenge
due to occlusions and complex object arrangements. We have developed
ThinkGrasp, a plug-and-play vision-language grasping system that makes use of
GPT-4o's advanced contextual reasoning for heavy clutter environment grasping
strategies. ThinkGrasp can effectively identify and generate grasp poses for
target objects, even when they are heavily obstructed or nearly invisible, by
using goal-oriented language to guide the removal of obstructing objects. This
approach progressively uncovers the target object and ultimately grasps it with
a few steps and a high success rate. In both simulated and real experiments,
ThinkGrasp achieved a high success rate and significantly outperformed
state-of-the-art methods in heavily cluttered environments or with diverse
unseen objects, demonstrating strong generalization capabilities.
Practical Unlearning for Large Language Models
Authors: name: Chongyang Gao, name: Lixu Wang, name: Chenkai Weng, name: Xiao Wang, name: Qi Zhu
Published: 2024-07-14 14:26:17

While LLMs have demonstrated impressive performance across various domains
and tasks, their security issues have become increasingly severe. Machine
unlearning (MU) has emerged as a promising solution to address these issues by
removing the influence of undesired data on the target model without
compromising its utility in other aspects. MU typically assumes full access to
the original training data to preserve utility, which is difficult to achieve
in LLM unlearning. Existing LLM unlearning methods often assume access to data
most affected by undesired data unlearning. However, this assumption
underestimates the entanglement among various LLM capabilities and ignores data
access limitations due to various issues. Moreover, these LLM unlearning
methods do not sufficiently consider that unlearning requests in real-world
scenarios are continuously emerging. To overcome these challenges and achieve
practical LLM unlearning, we propose the O3 framework. The O3 framework
includes an Out-Of-Distribution (OOD) detector to measure the similarity
between input and unlearning data, and an Orthogonal low-rank adapter (LoRA)
for continuously unlearning requested data. The OOD detector is trained with a
novel contrastive entropy loss and utilizes a local-global layer-aggregated
scoring mechanism. The orthogonal LoRA achieves parameter disentanglement among
continual unlearning requests. During inference, our O3 framework can smartly
decide whether and to what extent to load the unlearning LoRA based on the OOD
detector's predictions. Notably, O3's effectiveness does not rely on any
retained data. We conducted extensive experiments on O3 and state-of-the-art
LLM unlearning methods across three tasks and seven datasets. The results
indicate that O3 consistently achieves the best trade-off between unlearning
effectiveness and utility preservation, especially when facing continuous
unlearning requests.
Zero-shot Cross-Lingual Transfer for Synthetic Data Generation in
  Grammatical Error Detection
Authors: name: Gaetan Lopez Latouche, name: Marc-André Carbonneau, name: Ben Swanson
Published: 2024-07-16 15:35:15

Grammatical Error Detection (GED) methods rely heavily on human annotated
error corpora. However, these annotations are unavailable in many low-resource
languages. In this paper, we investigate GED in this context. Leveraging the
zero-shot cross-lingual transfer capabilities of multilingual pre-trained
language models, we train a model using data from a diverse set of languages to
generate synthetic errors in other languages. These synthetic error corpora are
then used to train a GED model. Specifically we propose a two-stage fine-tuning
pipeline where the GED model is first fine-tuned on multilingual synthetic data
from target languages followed by fine-tuning on human-annotated GED corpora
from source languages. This approach outperforms current state-of-the-art
annotation-free GED methods. We also analyse the errors produced by our method
and other strong baselines, finding that our approach produces errors that are
more diverse and more similar to human errors.
AUITestAgent: Automatic Requirements Oriented GUI Function Testing
Authors: name: Yongxiang Hu, name: Xuan Wang, name: Yingchuan Wang, name: Yu Zhang, name: Shiyu Guo, name: Chaoyi Chen, name: Xin Wang, name: Yangfan Zhou
Published: 2024-07-12 06:14:46

The Graphical User Interface (GUI) is how users interact with mobile apps. To
ensure it functions properly, testing engineers have to make sure it functions
as intended, based on test requirements that are typically written in natural
language. While widely adopted manual testing and script-based methods are
effective, they demand substantial effort due to the vast number of GUI pages
and rapid iterations in modern mobile apps. This paper introduces AUITestAgent,
the first automatic, natural language-driven GUI testing tool for mobile apps,
capable of fully automating the entire process of GUI interaction and function
verification. Since test requirements typically contain interaction commands
and verification oracles. AUITestAgent can extract GUI interactions from test
requirements via dynamically organized agents. Then, AUITestAgent employs a
multi-dimensional data extraction strategy to retrieve data relevant to the
test requirements from the interaction trace and perform verification.
Experiments on customized benchmarks demonstrate that AUITestAgent outperforms
existing tools in the quality of generated GUI interactions and achieved the
accuracy of verifications of 94%. Moreover, field deployment in Meituan has
shown AUITestAgent's practical usability, with it detecting 4 new functional
bugs during 10 regression tests in two months.
AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge
  Bases
Authors: name: Zhaorun Chen, name: Zhen Xiang, name: Chaowei Xiao, name: Dawn Song, name: Bo Li
Published: 2024-07-17 17:59:47

LLM agents have demonstrated remarkable performance across various
applications, primarily due to their advanced capabilities in reasoning,
utilizing external knowledge and tools, calling APIs, and executing actions to
interact with environments. Current agents typically utilize a memory module or
a retrieval-augmented generation (RAG) mechanism, retrieving past knowledge and
instances with similar embeddings from knowledge bases to inform task planning
and execution. However, the reliance on unverified knowledge bases raises
significant concerns about their safety and trustworthiness. To uncover such
vulnerabilities, we propose a novel red teaming approach AgentPoison, the first
backdoor attack targeting generic and RAG-based LLM agents by poisoning their
long-term memory or RAG knowledge base. In particular, we form the trigger
generation process as a constrained optimization to optimize backdoor triggers
by mapping the triggered instances to a unique embedding space, so as to ensure
that whenever a user instruction contains the optimized backdoor trigger, the
malicious demonstrations are retrieved from the poisoned memory or knowledge
base with high probability. In the meantime, benign instructions without the
trigger will still maintain normal performance. Unlike conventional backdoor
attacks, AgentPoison requires no additional model training or fine-tuning, and
the optimized backdoor trigger exhibits superior transferability, in-context
coherence, and stealthiness. Extensive experiments demonstrate AgentPoison's
effectiveness in attacking three types of real-world LLM agents: RAG-based
autonomous driving agent, knowledge-intensive QA agent, and healthcare
EHRAgent. On each agent, AgentPoison achieves an average attack success rate
higher than 80% with minimal impact on benign performance (less than 1%) with a
poison rate less than 0.1%.
GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill
  and Extreme KV-Cache Compression
Authors: name: Daniel Goldstein, name: Fares Obeid, name: Eric Alcaide, name: Guangyu Song, name: Eugene Cheah
Published: 2024-07-16 18:00:00

We introduce GoldFinch, a hybrid Linear Attention/Transformer sequence model
that uses a new technique to efficiently generate a highly compressed and
reusable KV-Cache in linear time and space with respect to sequence length.
GoldFinch stacks our new GOLD transformer on top of an enhanced version of the
Finch (RWKV-6) architecture. We train up to 1.5B parameter class models of the
Finch, Llama, and GoldFinch architectures, and find dramatically improved
modeling performance relative to both Finch and Llama. Our cache size savings
increase linearly with model layer count, ranging from 756-2550 times smaller
than the traditional transformer cache for common sizes, enabling inference of
extremely large context lengths even on limited hardware. Although
autoregressive generation has O(n) time complexity per token because of
attention, pre-fill computation of the entire initial cache state for a
submitted context costs only O(1) time per token due to the use of a recurrent
neural network (RNN) to generate this cache. We release our trained weights and
training code under the Apache 2.0 license for community use.
Case2Code: Learning Inductive Reasoning with Synthetic Data
Authors: name: Yunfan Shao, name: Linyang Li, name: Yichuan Ma, name: Peiji Li, name: Demin Song, name: Qinyuan Cheng, name: Shimin Li, name: Xiaonan Li, name: Pengyu Wang, name: Qipeng Guo, name: Hang Yan, name: Xipeng Qiu, name: Xuanjing Huang, name: Dahua Lin
Published: 2024-07-17 11:35:00

Complex reasoning is an impressive ability shown by large language models
(LLMs). Most LLMs are skilled in deductive reasoning, such as chain-of-thought
prompting or iterative tool-using to solve challenging tasks step-by-step. In
this paper, we hope to focus on evaluating and teaching LLMs to conduct
inductive reasoning, that is, LLMs are supposed to infer underlying rules by
observing examples or sequential transformations. However, collecting
large-scale and diverse human-generated inductive data is challenging. We focus
on data synthesis in the code domain and propose a Case2Code task by
exploiting the expressiveness and correctness of programs. Specifically, we
collect a diverse set of executable programs, synthesize input-output
transformations for each program, and force LLMs to infer the underlying code
implementations based on the synthetic I/O cases. We first evaluate
representative LLMs on the synthesized Case2Code task and demonstrate that the
Case-to-code induction is challenging for LLMs. Then, we synthesize large-scale
Case2Code training samples to train LLMs to perform inductive reasoning.
Experimental results show that such induction training benefits not only in
distribution Case2Code performance but also enhances various coding abilities
of trained LLMs, demonstrating the great potential of learning inductive
reasoning via synthetic data.
Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language
  Models
Authors: name: Ayush Kaushal, name: Tejas Pandey, name: Tejas Vaidhya, name: Aaryan Bhagat, name: Irina Rish
Published: 2024-07-17 05:53:20

Post-training quantization is the leading method for addressing
memory-related bottlenecks in LLM inference, but unfortunately, it suffers from
significant performance degradation below 4-bit precision. An alternative
approach involves training compressed models directly at a low bitwidth (e.g.,
binary or ternary models). However, the performance, training dynamics, and
scaling trends of such models are not yet well understood. To address this
issue, we train and openly release the Spectra LLM suite consisting of 54
language models ranging from 99M to 3.9B parameters, trained on 300B tokens.
Spectra includes FloatLMs, post-training quantized QuantLMs (3, 4, 6, and 8
bits), and ternary LLMs (TriLMs) - our improved architecture for ternary
language modeling, which significantly outperforms previously proposed ternary
models of a given size (in bits), matching half-precision models at scale. For
example, TriLM 3.9B is (bit-wise) smaller than the half-precision FloatLM 830M,
but matches half-precision FloatLM 3.9B in commonsense reasoning and knowledge
benchmarks. However, TriLM 3.9B is also as toxic and stereotyping as FloatLM
3.9B, a model six times larger in size. Additionally, TriLM 3.9B lags behind
FloatLM in perplexity on validation splits and web-based corpora but performs
better on less noisy datasets like Lambada and PennTreeBank.
  To enhance understanding of low-bitwidth models, we are releasing 500+
intermediate checkpoints of the Spectra suite at
https://github.com/NolanoOrg/SpectraSuite{https://github.com/NolanoOrg/SpectraSuite}.
The Art of Saying No: Contextual Noncompliance in Language Models
Authors: name: Faeze Brahman, name: Sachin Kumar, name: Vidhisha Balachandran, name: Pradeep Dasigi, name: Valentina Pyatkin, name: Abhilasha Ravichander, name: Sarah Wiegreffe, name: Nouha Dziri, name: Khyathi Chandu, name: Jack Hessel, name: Yulia Tsvetkov, name: Noah A. Smith, name: Yejin Choi, name: Hannaneh Hajishirzi
Published: 2024-07-02 07:12:51

Chat-based language models are designed to be helpful, yet they should not
comply with every user request. While most existing work primarily focuses on
refusal of "unsafe" queries, we posit that the scope of noncompliance should be
broadened. We introduce a comprehensive taxonomy of contextual noncompliance
describing when and how models should not comply with user requests. Our
taxonomy spans a wide range of categories including incomplete, unsupported,
indeterminate, and humanizing requests (in addition to unsafe requests). To
test noncompliance capabilities of language models, we use this taxonomy to
develop a new evaluation suite of 1000 noncompliance prompts. We find that most
existing models show significantly high compliance rates in certain previously
understudied categories with models like GPT-4 incorrectly complying with as
many as 30% of requests. To address these gaps, we explore different training
strategies using a synthetically-generated training set of requests and
expected noncompliant responses. Our experiments demonstrate that while direct
finetuning of instruction-tuned models can lead to both over-refusal and a
decline in general capabilities, using parameter efficient methods like low
rank adapters helps to strike a good balance between appropriate noncompliance
and other capabilities.
IMAGDressing-v1: Customizable Virtual Dressing
Authors: name: Fei Shen, name: Xin Jiang, name: Xin He, name: Hu Ye, name: Cong Wang, name: Xiaoyu Du, name: Zechao Li, name: Jinghui Tang
Published: 2024-07-17 16:26:30

Latest advances have achieved realistic virtual try-on (VTON) through
localized garment inpainting using latent diffusion models, significantly
enhancing consumers' online shopping experience. However, existing VTON
technologies neglect the need for merchants to showcase garments
comprehensively, including flexible control over garments, optional faces,
poses, and scenes. To address this issue, we define a virtual dressing (VD)
task focused on generating freely editable human images with fixed garments and
optional conditions. Meanwhile, we design a comprehensive affinity metric index
(CAMI) to evaluate the consistency between generated images and reference
garments. Then, we propose IMAGDressing-v1, which incorporates a garment UNet
that captures semantic features from CLIP and texture features from VAE. We
present a hybrid attention module, including a frozen self-attention and a
trainable cross-attention, to integrate garment features from the garment UNet
into a frozen denoising UNet, ensuring users can control different scenes
through text. IMAGDressing-v1 can be combined with other extension plugins,
such as ControlNet and IP-Adapter, to enhance the diversity and controllability
of generated images. Furthermore, to address the lack of data, we release the
interactive garment pairing (IGPair) dataset, containing over 300,000 pairs of
clothing and dressed images, and establish a standard pipeline for data
assembly. Extensive experiments demonstrate that our IMAGDressing-v1 achieves
state-of-the-art human image synthesis performance under various controlled
conditions. The code and model will be available at
https://github.com/muzishen/IMAGDressing.
Goldfish: Vision-Language Understanding of Arbitrarily Long Videos
Authors: name: Kirolos Ataallah, name: Xiaoqian Shen, name: Eslam Abdelrahman, name: Essam Sleiman, name: Mingchen Zhuge, name: Jian Ding, name: Deyao Zhu, name: Jürgen Schmidhuber, name: Mohamed Elhoseiny
Published: 2024-07-17 15:59:32

Most current LLM-based models for video understanding can process videos
within minutes. However, they struggle with lengthy videos due to challenges
such as "noise and redundancy", as well as "memory and computation"
constraints. In this paper, we present Goldfish, a methodology tailored for
comprehending videos of arbitrary lengths. We also introduce the TVQA-long
benchmark, specifically designed to evaluate models' capabilities in
understanding long videos with questions in both vision and text content.
Goldfish approaches these challenges with an efficient retrieval mechanism that
initially gathers the top-k video clips relevant to the instruction before
proceeding to provide the desired response. This design of the retrieval
mechanism enables the Goldfish to efficiently process arbitrarily long video
sequences, facilitating its application in contexts such as movies or
television series. To facilitate the retrieval process, we developed
MiniGPT4-Video that generates detailed descriptions for the video clips. In
addressing the scarcity of benchmarks for long video evaluation, we adapted the
TVQA short video benchmark for extended content analysis by aggregating
questions from entire episodes, thereby shifting the evaluation from partial to
full episode comprehension. We attained a 41.78% accuracy rate on the TVQA-long
benchmark, surpassing previous methods by 14.94%. Our MiniGPT4-Video also shows
exceptional performance in short video comprehension, exceeding existing
state-of-the-art methods by 3.23%, 2.03%, 16.5% and 23.59% on the MSVD, MSRVTT,
TGIF, and TVQA short video benchmarks, respectively. These results indicate
that our models have significant improvements in both long and short-video
understanding. Our models and code have been made publicly available at
https://vision-cair.github.io/Goldfish_website/
Patch-Level Training for Large Language Models
Authors: name: Chenze Shao, name: Fandong Meng, name: Jie Zhou
Published: 2024-07-17 15:48:39

As Large Language Models (LLMs) achieve remarkable progress in language
understanding and generation, their training efficiency has become a critical
concern. Traditionally, LLMs are trained to predict the next token in a
sequence. Despite the success of token-level training, it suffers from
considerable computational costs due to the need to process an extensive number
of tokens. To mitigate this issue, this paper introduces patch-level training
for LLMs, which reduces the sequence length by compressing multiple tokens into
a single patch. During patch-level training, we feed the language model shorter
sequences of patches and train it to predict the next patch, thereby processing
the majority of the training data at a significantly reduced computational
cost. Following this, the model continues token-level training on the remaining
training data to align with the inference mode. Experiments on a diverse range
of models (370M-2.7B parameters) demonstrate that patch-level training can
reduce overall computational costs to 0.5times, without compromising the
model performance compared to token-level training. Source code:
https://github.com/shaochenze/PatchTrain.
E5-V: Universal Embeddings with Multimodal Large Language Models
Authors: name: Ting Jiang, name: Minghui Song, name: Zihan Zhang, name: Haizhen Huang, name: Weiwei Deng, name: Feng Sun, name: Qi Zhang, name: Deqing Wang, name: Fuzhen Zhuang
Published: 2024-07-17 14:04:12

Multimodal large language models (MLLMs) have shown promising advancements in
general visual and language understanding. However, the representation of
multimodal information using MLLMs remains largely unexplored. In this work, we
introduce a new framework, E5-V, designed to adapt MLLMs for achieving
universal multimodal embeddings. Our findings highlight the significant
potential of MLLMs in representing multimodal inputs compared to previous
approaches. By leveraging MLLMs with prompts, E5-V effectively bridges the
modality gap between different types of inputs, demonstrating strong
performance in multimodal embeddings even without fine-tuning. We propose a
single modality training approach for E5-V, where the model is trained
exclusively on text pairs. This method demonstrates significant improvements
over traditional multimodal training on image-text pairs, while reducing
training costs by approximately 95%. Additionally, this approach eliminates the
need for costly multimodal training data collection. Extensive experiments
across four types of tasks demonstrate the effectiveness of E5-V. As a
universal multimodal model, E5-V not only achieves but often surpasses
state-of-the-art performance in each task, despite being trained on a single
modality.