Orionnnn/hub-stats huggingface dataset:
NL-Eye: Abductive NLI for Images
Authors: name: Mor Ventura, name: Michael Toker, name: Nitay Calderon, name: Zorik Gekhman, name: Yonatan Bitton, name: Roi Reichart
Published: 2024-10-03 15:51:36

Will a Visual Language Model (VLM)-based bot warn us about slipping if it
detects a wet floor? Recent VLMs have demonstrated impressive capabilities, yet
their ability to infer outcomes and causes remains underexplored. To address
this, we introduce NL-Eye, a benchmark designed to assess VLMs' visual
abductive reasoning skills. NL-Eye adapts the abductive Natural Language
Inference (NLI) task to the visual domain, requiring models to evaluate the
plausibility of hypothesis images based on a premise image and explain their
decisions. NL-Eye consists of 350 carefully curated triplet examples (1,050
images) spanning diverse reasoning categories: physical, functional, logical,
emotional, cultural, and social. The data curation process involved two steps -
writing textual descriptions and generating images using text-to-image models,
both requiring substantial human involvement to ensure high-quality and
challenging scenes. Our experiments show that VLMs struggle significantly on
NL-Eye, often performing at random baseline levels, while humans excel in both
plausibility prediction and explanation quality. This demonstrates a deficiency
in the abductive reasoning capabilities of modern VLMs. NL-Eye represents a
crucial step toward developing VLMs capable of robust multimodal reasoning for
real-world applications, including accident-prevention bots and generated video
verification.
CANVAS: Commonsense-Aware Navigation System for Intuitive Human-Robot
  Interaction
Authors: name: Suhwan Choi, name: Yongjun Cho, name: Minchan Kim, name: Jaeyoon Jung, name: Myunchul Joe, name: Yubeen Park, name: Minseo Kim, name: Sungwoong Kim, name: Sungjae Lee, name: Hwiseong Park, name: Jiwan Chung, name: Youngjae Yu
Published: 2024-10-02 06:34:45

Real-life robot navigation involves more than just reaching a destination; it
requires optimizing movements while addressing scenario-specific goals. An
intuitive way for humans to express these goals is through abstract cues like
verbal commands or rough sketches. Such human guidance may lack details or be
noisy. Nonetheless, we expect robots to navigate as intended. For robots to
interpret and execute these abstract instructions in line with human
expectations, they must share a common understanding of basic navigation
concepts with humans. To this end, we introduce CANVAS, a novel framework that
combines visual and linguistic instructions for commonsense-aware navigation.
Its success is driven by imitation learning, enabling the robot to learn from
human navigation behavior. We present COMMAND, a comprehensive dataset with
human-annotated navigation results, spanning over 48 hours and 219 km, designed
to train commonsense-aware navigation systems in simulated environments. Our
experiments show that CANVAS outperforms the strong rule-based system ROS
NavStack across all environments, demonstrating superior performance with noisy
instructions. Notably, in the orchard environment, where ROS NavStack records a
0% total success rate, CANVAS achieves a total success rate of 67%. CANVAS also
closely aligns with human demonstrations and commonsense constraints, even in
unseen environments. Furthermore, real-world deployment of CANVAS showcases
impressive Sim2Real transfer with a total success rate of 69%, highlighting the
potential of learning from human demonstrations in simulated environments for
real-world applications.
A Comprehensive Survey of Mamba Architectures for Medical Image
  Analysis: Classification, Segmentation, Restoration and Beyond
Authors: name: Shubhi Bansal, name: Sreeharish A, name: Madhava Prasath J, name: Manikandan S, name: Sreekanth Madisetty, name: Mohammad Zia Ur Rehman, name: Chandravardhan Singh Raghaw, name: Gaurav Duggal, name: Nagendra Kumar
Published: 2024-10-03 10:23:03

Mamba, a special case of the State Space Model, is gaining popularity as an
alternative to template-based deep learning approaches in medical image
analysis. While transformers are powerful architectures, they have drawbacks,
including quadratic computational complexity and an inability to address
long-range dependencies efficiently. This limitation affects the analysis of
large and complex datasets in medical imaging, where there are many spatial and
temporal relationships. In contrast, Mamba offers benefits that make it
well-suited for medical image analysis. It has linear time complexity, which is
a significant improvement over transformers. Mamba processes longer sequences
without attention mechanisms, enabling faster inference and requiring less
memory. Mamba also demonstrates strong performance in merging multimodal data,
improving diagnosis accuracy and patient outcomes. The organization of this
paper allows readers to appreciate the capabilities of Mamba in medical imaging
step by step. We begin by defining core concepts of SSMs and models, including
S4, S5, and S6, followed by an exploration of Mamba architectures such as pure
Mamba, U-Net variants, and hybrid models with convolutional neural networks,
transformers, and Graph Neural Networks. We also cover Mamba optimizations,
techniques and adaptations, scanning, datasets, applications, experimental
results, and conclude with its challenges and future directions in medical
imaging. This review aims to demonstrate the transformative potential of Mamba
in overcoming existing barriers within medical imaging while paving the way for
innovative advancements in the field. A comprehensive list of Mamba
architectures applied in the medical field, reviewed in this work, is available
at Github.
RoCoTex: A Robust Method for Consistent Texture Synthesis with Diffusion
  Models
Authors: name: Jangyeong Kim, name: Donggoo Kang, name: Junyoung Choi, name: Jeonga Wi, name: Junho Gwon, name: Jiun Bae, name: Dumim Yoon, name: Junghyun Han
Published: 2024-09-30 06:29:50

Text-to-texture generation has recently attracted increasing attention, but
existing methods often suffer from the problems of view inconsistencies,
apparent seams, and misalignment between textures and the underlying mesh. In
this paper, we propose a robust text-to-texture method for generating
consistent and seamless textures that are well aligned with the mesh. Our
method leverages state-of-the-art 2D diffusion models, including SDXL and
multiple ControlNets, to capture structural features and intricate details in
the generated textures. The method also employs a symmetrical view synthesis
strategy combined with regional prompts for enhancing view consistency.
Additionally, it introduces novel texture blending and soft-inpainting
techniques, which significantly reduce the seam regions. Extensive experiments
demonstrate that our method outperforms existing state-of-the-art methods.
MIGA: Mixture-of-Experts with Group Aggregation for Stock Market
  Prediction
Authors: name: Zhaojian Yu, name: Yinghao Wu, name: Genesis Wang, name: Heming Weng
Published: 2024-10-03 06:26:49

Stock market prediction has remained an extremely challenging problem for
many decades owing to its inherent high volatility and low information noisy
ratio. Existing solutions based on machine learning or deep learning
demonstrate superior performance by employing a single model trained on the
entire stock dataset to generate predictions across all types of stocks.
However, due to the significant variations in stock styles and market trends, a
single end-to-end model struggles to fully capture the differences in these
stylized stock features, leading to relatively inaccurate predictions for all
types of stocks. In this paper, we present MIGA, a novel Mixture of Expert with
Group Aggregation framework designed to generate specialized predictions for
stocks with different styles by dynamically switching between distinct style
experts. To promote collaboration among different experts in MIGA, we propose a
novel inner group attention architecture, enabling experts within the same
group to share information and thereby enhancing the overall performance of all
experts. As a result, MIGA significantly outperforms other end-to-end models on
three Chinese Stock Index benchmarks including CSI300, CSI500, and CSI1000.
Notably, MIGA-Conv reaches 24 % excess annual return on CSI300 benchmark,
surpassing the previous state-of-the-art model by 8% absolute. Furthermore, we
conduct a comprehensive analysis of mixture of experts for stock market
prediction, providing valuable insights for future research.
Addition is All You Need for Energy-efficient Language Models
Authors: name: Hongyin Luo, name: Wei Sun
Published: 2024-10-01 17:53:28

Large neural networks spend most computation on floating point tensor
multiplications. In this work, we find that a floating point multiplier can be
approximated by one integer adder with high precision. We propose the
linear-complexity multiplication L-Mul algorithm that approximates floating
point number multiplication with integer addition operations. The new algorithm
costs significantly less computation resource than 8-bit floating point
multiplication but achieves higher precision. Compared to 8-bit floating point
multiplications, the proposed method achieves higher precision but consumes
significantly less bit-level computation. Since multiplying floating point
numbers requires substantially higher energy compared to integer addition
operations, applying the L-Mul operation in tensor processing hardware can
potentially reduce 95% energy cost by element-wise floating point tensor
multiplications and 80% energy cost of dot products. We calculated the
theoretical error expectation of L-Mul, and evaluated the algorithm on a wide
range of textual, visual, and symbolic tasks, including natural language
understanding, structural reasoning, mathematics, and commonsense question
answering. Our numerical analysis experiments agree with the theoretical error
estimation, which indicates that L-Mul with 4-bit mantissa achieves comparable
precision as float8_e4m3 multiplications, and L-Mul with 3-bit mantissa
outperforms float8_e5m2. Evaluation results on popular benchmarks show that
directly applying L-Mul to the attention mechanism is almost lossless. We
further show that replacing all floating point multiplications with 3-bit
mantissa L-Mul in a transformer model achieves equivalent precision as using
float8_e4m3 as accumulation precision in both fine-tuning and inference.
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Authors: name: Rose E. Wang, name: Ana T. Ribeiro, name: Carly D. Robinson, name: Susanna Loeb, name: Dora Demszky
Published: 2024-10-03 21:58:39

Generative AI, particularly Language Models (LMs), has the potential to
transform real-world domains with societal impact, particularly where access to
experts is limited. For example, in education, training novice educators with
expert guidance is important for effectiveness but expensive, creating
significant barriers to improving education quality at scale. This challenge
disproportionately harms students from under-served communities, who stand to
gain the most from high-quality education. We introduce Tutor CoPilot, a novel
Human-AI approach that leverages a model of expert thinking to provide
expert-like guidance to tutors as they tutor. This study is the first
randomized controlled trial of a Human-AI system in live tutoring, involving
900 tutors and 1,800 K-12 students from historically under-served communities.
Following a preregistered analysis plan, we find that students working with
tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more
likely to master topics (p<0.01). Notably, students of lower-rated tutors
experienced the greatest benefit, improving mastery by 9 p.p. We find that
Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages
using classifiers to identify pedagogical strategies, and find that tutors with
access to Tutor CoPilot are more likely to use high-quality strategies to
foster student understanding (e.g., asking guiding questions) and less likely
to give away the answer to the student. Tutor interviews highlight how Tutor
CoPilot's guidance helps tutors to respond to student needs, though they flag
issues in Tutor CoPilot, such as generating suggestions that are not
grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates
how Human-AI systems can scale expertise in real-world domains, bridge gaps in
skills and create a future where high-quality education is accessible to all
students.
Erasing Conceptual Knowledge from Language Models
Authors: name: Rohit Gandikota, name: Sheridan Feucht, name: Samuel Marks, name: David Bau
Published: 2024-10-03 17:59:30

Concept erasure in language models has traditionally lacked a comprehensive
evaluation framework, leading to incomplete assessments of effectiveness of
erasure methods. We propose an evaluation paradigm centered on three critical
criteria: innocence (complete knowledge removal), seamlessness (maintaining
conditional fluent generation), and specificity (preserving unrelated task
performance). Our evaluation metrics naturally motivate the development of
Erasure of Language Memory (ELM), a new method designed to address all three
dimensions. ELM employs targeted low-rank updates to alter output distributions
for erased concepts while preserving overall model capabilities including
fluency when prompted for an erased concept. We demonstrate ELM's efficacy on
biosecurity, cybersecurity, and literary domain erasure tasks. Comparative
analysis shows that ELM achieves superior performance across our proposed
metrics, including near-random scores on erased topic assessments, generation
fluency, maintained accuracy on unrelated benchmarks, and robustness under
adversarial attacks. Our code, data, and trained models are available at
https://elm.baulab.info
Selective Attention Improves Transformer
Authors: name: Yaniv Leviathan, name: Matan Kalman, name: Yossi Matias
Published: 2024-10-03 17:27:30

Unneeded elements in the attention's context degrade performance. We
introduce Selective Attention, a simple parameter-free change to the standard
attention mechanism which reduces attention to unneeded elements. Selective
attention improves language modeling performance in a variety of model sizes
and context lengths. For example, a range of transformers trained with the
language modeling objective on C4 with selective attention perform equivalently
to standard transformers with ~2X more heads and parameters in their attention
modules. Selective attention also allows decreasing the size of the attention's
context buffer, leading to meaningful reductions in the memory and compute
requirements during inference. For example, transformers with 100M parameters
trained on C4 with context sizes of 512, 1,024, and 2,048 need 16X, 25X, and
47X less memory for their attention module, respectively, when equipped with
selective attention, as those without selective attention, with the same
validation perplexity.
Contextual Document Embeddings
Authors: name: John X. Morris, name: Alexander M. Rush
Published: 2024-10-03 14:33:34

Dense document embeddings are central to neural retrieval. The dominant
paradigm is to train and construct embeddings by running encoders directly on
individual documents. In this work, we argue that these embeddings, while
effective, are implicitly out-of-context for targeted use cases of retrieval,
and that a contextualized document embedding should take into account both the
document and neighboring documents in context - analogous to contextualized
word embeddings. We propose two complementary methods for contextualized
document embeddings: first, an alternative contrastive learning objective that
explicitly incorporates the document neighbors into the intra-batch contextual
loss; second, a new contextual architecture that explicitly encodes neighbor
document information into the encoded representation. Results show that both
methods achieve better performance than biencoders in several settings, with
differences especially pronounced out-of-domain. We achieve state-of-the-art
results on the MTEB benchmark with no hard negative mining, score distillation,
dataset-specific instructions, intra-GPU example-sharing, or extremely large
batch sizes. Our method can be applied to improve performance on any
contrastive learning dataset and any biencoder.
Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language
  Models
Authors: name: Lucas Bandarkar, name: Benjamin Muller, name: Pritish Yuvraj, name: Rui Hou, name: Nayan Singhal, name: Hongjiang Lv, name: Bing Liu
Published: 2024-10-02 08:53:07

Model merging, such as model souping, is the practice of combining different
models with the same architecture together without further training. In this
work, we present a model merging methodology that addresses the difficulty of
fine-tuning Large Language Models (LLMs) for target tasks in non-English
languages, where task-specific data is often unavailable. We focus on
mathematical reasoning and without in-language math data, facilitate
cross-lingual transfer by composing language and math capabilities. Starting
from the same pretrained model, we fine-tune separate "experts" on math
instruction data in English and on generic instruction data in the target
language. We then replace the top and bottom transformer layers of the math
expert directly with layers from the language expert, which consequently
enhances math performance in the target language. The resulting merged models
outperform the individual experts and other merging methods on the math
benchmark, MGSM, by 10% across four major languages where math instruction data
is scarce. In addition, this layer swapping is simple, inexpensive, and
intuitive, as it is based on an interpretative analysis of the most important
parameter changes during the fine-tuning of each expert. The ability to
successfully re-compose LLMs for cross-lingual transfer in this manner opens up
future possibilities to combine model expertise, create modular solutions, and
transfer reasoning capabilities across languages all post hoc.
Interpreting and Editing Vision-Language Representations to Mitigate
  Hallucinations
Authors: name: Nick Jiang, name: Anish Kachinthaya, name: Suzie Petryk, name: Yossi Gandelsman
Published: 2024-10-03 17:59:57

We investigate the internal representations of vision-language models (VLMs)
to address hallucinations, a persistent challenge despite advances in model
size and training. We project VLMs' internal image representations to their
language vocabulary and observe more confident output probabilities on real
objects than hallucinated objects. We additionally use these output
probabilities to spatially localize real objects. Building on this approach, we
introduce a knowledge erasure algorithm that removes hallucinations by linearly
orthogonalizing image features with respect to hallucinated object features. We
show that targeted edits to a model's latent representations can reduce
hallucinations by up to 25.7% on the COCO2014 dataset while preserving
performance. Our findings demonstrate how a deeper understanding of VLMs'
latent representations can enhance reliability and enable novel capabilities,
such as zero-shot segmentation.
Improving Autonomous AI Agents with Reflective Tree Search and
  Self-Learning
Authors: name: Xiao Yu, name: Baolin Peng, name: Vineeth Vajipey, name: Hao Cheng, name: Michel Galley, name: Jianfeng Gao, name: Zhou Yu
Published: 2024-10-02 21:42:35

Autonomous agents have demonstrated significant potential in automating
complex multistep decision-making tasks. However, even state-of-the-art
vision-language models (VLMs), such as GPT-4o, still fall short of human-level
performance, particularly in intricate web environments and long-horizon
planning tasks. To address these limitations, we introduce Reflective Monte
Carlo Tree Search (R-MCTS), a novel test-time algorithm designed to enhance the
ability of AI agents, e.g., powered by GPT-4o, to explore decision space on the
fly. R-MCTS extends traditional MCTS by 1) incorporating contrastive
reflection, allowing agents to learn from past interactions and dynamically
improve their search efficiency; and 2) using multi-agent debate to provide
reliable state evaluation. Moreover, we improve the agent's performance by
fine-tuning GPT-4o through self-learning, using R-MCTS generated tree
traversals without any human-provided labels. On the challenging VisualWebArena
benchmark, our GPT-4o-based R-MCTS agent achieves a 6% to 30% relative
improvement across various tasks compared to the previous state-of-the-art.
Additionally, we show that the knowledge gained from test-time search can be
effectively transferred back to GPT-4o via fine-tuning. The fine-tuned GPT-4o
matches 97% of R-MCTS's performance while reducing compute usage by a factor of
four at test time. Furthermore, qualitative results reveal that the fine-tuned
GPT-4o model demonstrates the ability to explore the environment, evaluate a
state, and backtrack to viable ones when it detects that the current state
cannot lead to success. Moreover, our work demonstrates the compute scaling
properties in both training - data collection with R-MCTS - and testing time.
These results suggest a promising research direction to enhance VLMs' reasoning
and planning capabilities for agentic applications via test-time search and
self-learning.
Open-RAG: Enhanced Retrieval-Augmented Reasoning with Open-Source Large
  Language Models
Authors: name: Shayekh Bin Islam, name: Md Asib Rahman, name: K S M Tozammel Hossain, name: Enamul Hoque, name: Shafiq Joty, name: Md Rizwan Parvez
Published: 2024-10-02 17:37:18

Retrieval-Augmented Generation (RAG) has been shown to enhance the factual
accuracy of Large Language Models (LLMs), but existing methods often suffer
from limited reasoning capabilities in effectively using the retrieved
evidence, particularly when using open-source LLMs. To mitigate this gap, we
introduce a novel framework, Open-RAG, designed to enhance reasoning
capabilities in RAG with open-source LLMs. Our framework transforms an
arbitrary dense LLM into a parameter-efficient sparse mixture of experts (MoE)
model capable of handling complex reasoning tasks, including both single- and
multi-hop queries. Open-RAG uniquely trains the model to navigate challenging
distractors that appear relevant but are misleading. As a result, Open-RAG
leverages latent learning, dynamically selecting relevant experts and
integrating external knowledge effectively for more accurate and contextually
relevant responses. In addition, we propose a hybrid adaptive retrieval method
to determine retrieval necessity and balance the trade-off between performance
gain and inference speed. Experimental results show that the Llama2-7B-based
Open-RAG outperforms state-of-the-art LLMs and RAG models such as ChatGPT,
Self-RAG, and Command R+ in various knowledge-intensive tasks. We open-source
our code and models at https://openragmoe.github.io/