Orionnnn/hub-stats huggingface dataset:
Integrating Large Language Models into a Tri-Modal Architecture for
  Automated Depression Classification
Authors: name: Santosh V. Patapati
Published: 2024-07-27 21:00:36

Major Depressive Disorder (MDD) is a pervasive mental health condition that
affects 300 million people worldwide. This work presents a novel, BiLSTM-based
tri-modal model-level fusion architecture for the binary classification of
depression from clinical interview recordings. The proposed architecture
incorporates Mel Frequency Cepstral Coefficients, Facial Action Units, and uses
a two-shot learning based GPT-4 model to process text data. This is the first
work to incorporate large language models into a multi-modal architecture for
this task. It achieves impressive results on the DAIC-WOZ AVEC 2016 Challenge
cross-validation split and Leave-One-Subject-Out cross-validation split,
surpassing all baseline models and multiple state-of-the-art models. In
Leave-One-Subject-Out testing, it achieves an accuracy of 91.01%, an F1-Score
of 85.95%, a precision of 80%, and a recall of 92.86%.
ImagiNet: A Multi-Content Dataset for Generalizable Synthetic Image
  Detection via Contrastive Learning
Authors: name: Delyan Boychev, name: Radostin Cholakov
Published: 2024-07-29 13:57:24

Generative models, such as diffusion models (DMs), variational autoencoders
(VAEs), and generative adversarial networks (GANs), produce images with a level
of authenticity that makes them nearly indistinguishable from real photos and
artwork. While this capability is beneficial for many industries, the
difficulty of identifying synthetic images leaves online media platforms
vulnerable to impersonation and misinformation attempts. To support the
development of defensive methods, we introduce ImagiNet, a high-resolution and
balanced dataset for synthetic image detection, designed to mitigate potential
biases in existing resources. It contains 200K examples, spanning four content
categories: photos, paintings, faces, and uncategorized. Synthetic images are
produced with open-source and proprietary generators, whereas real counterparts
of the same content type are collected from public datasets. The structure of
ImagiNet allows for a two-track evaluation system: i) classification as real or
synthetic and ii) identification of the generative model. To establish a
baseline, we train a ResNet-50 model using a self-supervised contrastive
objective (SelfCon) for each track. The model demonstrates state-of-the-art
performance and high inference speed across established benchmarks, achieving
an AUC of up to 0.99 and balanced accuracy ranging from 86% to 95%, even under
social network conditions that involve compression and resizing. Our data and
code are available at https://github.com/delyan-boychev/imaginet.
Meta-Rewarding Language Models: Self-Improving Alignment with
  LLM-as-a-Meta-Judge
Authors: name: Tianhao Wu, name: Weizhe Yuan, name: Olga Golovneva, name: Jing Xu, name: Yuandong Tian, name: Jiantao Jiao, name: Jason Weston, name: Sainbayar Sukhbaatar
Published: 2024-07-28 21:58:28

Large Language Models (LLMs) are rapidly surpassing human knowledge in many
domains. While improving these models traditionally relies on costly human
data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs
can improve by judging their own responses instead of relying on human
labelers. However, existing methods have primarily focused on improving model
responses rather than judgment capabilities, resulting in rapid saturation
during iterative training. To address this issue, we introduce a novel
Meta-Rewarding step to the self-improvement process, where the model judges its
own judgements and uses that feedback to refine its judgment skills.
Surprisingly, this unsupervised approach improves the model's ability to judge
{\em and} follow instructions, as demonstrated by a win rate improvement of
Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on
Arena-Hard. These results strongly suggest the potential for self-improving
models without human supervision.
TAPTRv2: Attention-based Position Update Improves Tracking Any Point
Authors: name: Hongyang Li, name: Hao Zhang, name: Shilong Liu, name: Zhaoyang Zeng, name: Feng Li, name: Tianhe Ren, name: Bohan Li, name: Lei Zhang
Published: 2024-07-23 08:46:14

In this paper, we present TAPTRv2, a Transformer-based approach built upon
TAPTR for solving the Tracking Any Point (TAP) task. TAPTR borrows designs from
DEtection TRansformer (DETR) and formulates each tracking point as a point
query, making it possible to leverage well-studied operations in DETR-like
algorithms. TAPTRv2 improves TAPTR by addressing a critical issue regarding its
reliance on cost-volume,which contaminates the point query\'s content feature
and negatively impacts both visibility prediction and cost-volume computation.
In TAPTRv2, we propose a novel attention-based position update (APU) operation
and use key-aware deformable attention to realize. For each query, this
operation uses key-aware attention weights to combine their corresponding
deformable sampling positions to predict a new query position. This design is
based on the observation that local attention is essentially the same as
cost-volume, both of which are computed by dot-production between a query and
its surrounding features. By introducing this new operation, TAPTRv2 not only
removes the extra burden of cost-volume computation, but also leads to a
substantial performance improvement. TAPTRv2 surpasses TAPTR and achieves
state-of-the-art performance on many challenging datasets, demonstrating the
superiority
SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models
  for Southeast Asian Languages
Authors: name: Wenxuan Zhang, name: Hou Pong Chan, name: Yiran Zhao, name: Mahani Aljunied, name: Jianyu Wang, name: Chaoqun Liu, name: Yue Deng, name: Zhiqiang Hu, name: Weiwen Xu, name: Yew Ken Chia, name: Xin Li, name: Lidong Bing
Published: 2024-07-29 03:26:22

Large Language Models (LLMs) have shown remarkable abilities across various
tasks, yet their development has predominantly centered on high-resource
languages like English and Chinese, leaving low-resource languages underserved.
To address this disparity, we present SeaLLMs 3, the latest iteration of the
SeaLLMs model family, tailored for Southeast Asian languages. This region,
characterized by its rich linguistic diversity, has lacked adequate language
technology support. SeaLLMs 3 aims to bridge this gap by covering a
comprehensive range of languages spoken in this region, including English,
Chinese, Indonesian, Vietnamese, Thai, Tagalog, Malay, Burmese, Khmer, Lao,
Tamil, and Javanese. Leveraging efficient language enhancement techniques and a
specially constructed instruction tuning dataset, SeaLLMs 3 significantly
reduces training costs while maintaining high performance and versatility. Our
model excels in tasks such as world knowledge, mathematical reasoning,
translation, and instruction following, achieving state-of-the-art performance
among similarly sized models. Additionally, we prioritized safety and
reliability by addressing both general and culture-specific considerations and
incorporated mechanisms to reduce hallucinations. This work underscores the
importance of inclusive AI, showing that advanced LLM capabilities can benefit
underserved linguistic and cultural communities.
ATHAR: A High-Quality and Diverse Dataset for Classical Arabic to
  English Translation
Authors: name: Mohammed Khalil, name: Mohammed Sabry
Published: 2024-07-29 09:45:34

Classical Arabic represents a significant era, encompassing the golden age of
Arab culture, philosophy, and scientific literature. With a broad consensus on
the importance of translating these literatures to enrich knowledge
dissemination across communities, the advent of large language models (LLMs)
and translation systems offers promising tools to facilitate this goal.
However, we have identified a scarcity of translation datasets in Classical
Arabic, which are often limited in scope and topics, hindering the development
of high-quality translation systems. In response, we present the ATHAR dataset,
comprising 66,000 high-quality Classical Arabic to English translation samples
that cover a wide array of subjects including science, culture, and philosophy.
Furthermore, we assess the performance of current state-of-the-art LLMs under
various settings, concluding that there is a need for such datasets in current
systems. Our findings highlight how models can benefit from fine-tuning or
incorporating this dataset into their pretraining pipelines. The dataset is
publicly available on the HuggingFace Data Hub at
https://huggingface.co/datasets/mohamed-khalil/ATHAR.
Sentiment Analysis of Lithuanian Online Reviews Using Large Language
  Models
Authors: name: Brigita Vileikytė, name: Mantas Lukoševičius, name: Lukas Stankevičius
Published: 2024-07-29 11:44:21

Sentiment analysis is a widely researched area within Natural Language
Processing (NLP), attracting significant interest due to the advent of
automated solutions. Despite this, the task remains challenging because of the
inherent complexity of languages and the subjective nature of sentiments. It is
even more challenging for less-studied and less-resourced languages such as
Lithuanian. Our review of existing Lithuanian NLP research reveals that
traditional machine learning methods and classification algorithms have limited
effectiveness for the task. In this work, we address sentiment analysis of
Lithuanian five-star-based online reviews from multiple domains that we collect
and clean. We apply transformer models to this task for the first time,
exploring the capabilities of pre-trained multilingual Large Language Models
(LLMs), specifically focusing on fine-tuning BERT and T5 models. Given the
inherent difficulty of the task, the fine-tuned models perform quite well,
especially when the sentiments themselves are less ambiguous: 80.74% and 89.61%
testing recognition accuracy of the most popular one- and five-star reviews
respectively. They significantly outperform current commercial state-of-the-art
general-purpose LLM GPT-4. We openly share our fine-tuned LLMs online.
Mixture of Nested Experts: Adaptive Processing of Visual Tokens
Authors: name: Gagan Jain, name: Nidhi Hegde, name: Aditya Kusupati, name: Arsha Nagrani, name: Shyamal Buch, name: Prateek Jain, name: Anurag Arnab, name: Sujoy Paul
Published: 2024-07-29 13:19:31

The visual medium (images and videos) naturally contains a large amount of
information redundancy, thereby providing a great opportunity for leveraging
efficiency in processing. While Vision Transformer (ViT) based models scale
effectively to large data regimes, they fail to capitalize on this inherent
redundancy, leading to higher computational costs. Mixture of Experts (MoE)
networks demonstrate scalability while maintaining same inference-time costs,
but they come with a larger parameter footprint. We present Mixture of Nested
Experts (MoNE), which utilizes a nested structure for experts, wherein
individual experts fall on an increasing compute-accuracy curve. Given a
compute budget, MoNE learns to dynamically choose tokens in a priority order,
and thus redundant tokens are processed through cheaper nested experts. Using
this framework, we achieve equivalent performance as the baseline models, while
reducing inference time compute by over two-fold. We validate our approach on
standard image and video datasets - ImageNet-21K, Kinetics400, and
Something-Something-v2. We further highlight MoNE's adaptability by
showcasing its ability to maintain strong performance across different
inference-time compute budgets on videos, using only a single trained model.
MindSearch: Mimicking Human Minds Elicits Deep AI Searcher
Authors: name: Zehui Chen, name: Kuikun Liu, name: Qiuchen Wang, name: Jiangning Liu, name: Wenwei Zhang, name: Kai Chen, name: Feng Zhao
Published: 2024-07-29 17:12:40

Information seeking and integration is a complex cognitive task that consumes
enormous time and effort. Inspired by the remarkable progress of Large Language
Models, recent works attempt to solve this task by combining LLMs and search
engines. However, these methods still obtain unsatisfying performance due to
three challenges: (1) complex requests often cannot be accurately and
completely retrieved by the search engine once (2) corresponding information to
be integrated is spread over multiple web pages along with massive noise, and
(3) a large number of web pages with long contents may quickly exceed the
maximum context length of LLMs. Inspired by the cognitive process when humans
solve these problems, we introduce MindSearch to mimic the human minds in web
information seeking and integration, which can be instantiated by a simple yet
effective LLM-based multi-agent framework. The WebPlanner models the human mind
of multi-step information seeking as a dynamic graph construction process: it
decomposes the user query into atomic sub-questions as nodes in the graph and
progressively extends the graph based on the search result from WebSearcher.
Tasked with each sub-question, WebSearcher performs hierarchical information
retrieval with search engines and collects valuable information for WebPlanner.
The multi-agent design of MindSearch enables the whole framework to seek and
integrate information parallelly from larger-scale (e.g., more than 300) web
pages in 3 minutes, which is worth 3 hours of human effort. MindSearch
demonstrates significant improvement in the response quality in terms of depth
and breadth, on both close-set and open-set QA problems. Besides, responses
from MindSearch based on InternLM2.5-7B are preferable by humans to ChatGPT-Web
and Perplexity.ai applications, which implies that MindSearch can already
deliver a competitive solution to the proprietary AI search engine.
Diffusion Feedback Helps CLIP See Better
Authors: name: Wenxuan Wang, name: Quan Sun, name: Fan Zhang, name: Yepeng Tang, name: Jing Liu, name: Xinlong Wang
Published: 2024-07-29 17:00:09

Contrastive Language-Image Pre-training (CLIP), which excels at abstracting
open-world representations across domains and modalities, has become a
foundation for a variety of vision and multimodal tasks. However, recent
studies reveal that CLIP has severe visual shortcomings, such as which can
hardly distinguish orientation, quantity, color, structure, etc. These visual
shortcomings also limit the perception capabilities of multimodal large
language models (MLLMs) built on CLIP. The main reason could be that the
image-text pairs used to train CLIP are inherently biased, due to the lack of
the distinctiveness of the text and the diversity of images. In this work, we
present a simple post-training approach for CLIP models, which largely
overcomes its visual shortcomings via a self-supervised diffusion process. We
introduce DIVA, which uses the DIffusion model as a Visual Assistant for CLIP.
Specifically, DIVA leverages generative feedback from text-to-image diffusion
models to optimize CLIP representations, with only images (without
corresponding text). We demonstrate that DIVA improves CLIP's performance on
the challenging MMVP-VLM benchmark which assesses fine-grained visual abilities
to a large extent (e.g., 3-7%), and enhances the performance of MLLMs and
vision models on multimodal understanding and segmentation tasks. Extensive
evaluation on 29 image classification and retrieval benchmarks confirms that
our framework preserves CLIP's strong zero-shot capabilities. The code will be
available at https://github.com/baaivision/DIVA.
Bridging the Gap: Studio-like Avatar Creation from a Monocular Phone
  Capture
Authors: name: ShahRukh Athar, name: Shunsuke Saito, name: Zhengyu Yang, name: Stanislav Pidhorsky, name: Chen Cao
Published: 2024-07-28 21:26:33

Creating photorealistic avatars for individuals traditionally involves
extensive capture sessions with complex and expensive studio devices like the
LightStage system. While recent strides in neural representations have enabled
the generation of photorealistic and animatable 3D avatars from quick phone
scans, they have the capture-time lighting baked-in, lack facial details and
have missing regions in areas such as the back of the ears. Thus, they lag in
quality compared to studio-captured avatars. In this paper, we propose a method
that bridges this gap by generating studio-like illuminated texture maps from
short, monocular phone captures. We do this by parameterizing the phone texture
maps using the W^+ space of a StyleGAN2, enabling near-perfect
reconstruction. Then, we finetune a StyleGAN2 by sampling in the W^+
parameterized space using a very small set of studio-captured textures as an
adversarial training signal. To further enhance the realism and accuracy of
facial details, we super-resolve the output of the StyleGAN2 using carefully
designed diffusion model that is guided by image gradients of the
phone-captured texture map. Once trained, our method excels at producing
studio-like facial texture maps from casual monocular smartphone videos.
Demonstrating its capabilities, we showcase the generation of photorealistic,
uniformly lit, complete avatars from monocular phone captures.
http://shahrukhathar.github.io/2024/07/22/Bridging.html{The project page
can be found here.}
Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Authors: name: Jinghuan Shang, name: Karl Schmeckpeper, name: Brandon B. May, name: Maria Vittoria Minniti, name: Tarik Kelestemur, name: David Watkins, name: Laura Herlant
Published: 2024-07-29 17:08:21

Vision-based robot policy learning, which maps visual inputs to actions,
necessitates a holistic understanding of diverse visual tasks beyond
single-task needs like classification or segmentation. Inspired by this, we
introduce Theia, a vision foundation model for robot learning that distills
multiple off-the-shelf vision foundation models trained on varied vision tasks.
Theia's rich visual representations encode diverse visual knowledge, enhancing
downstream robot learning. Extensive experiments demonstrate that Theia
outperforms its teacher models and prior robot learning models using less
training data and smaller model sizes. Additionally, we quantify the quality of
pre-trained visual representations and hypothesize that higher entropy in
feature norm distributions leads to improved robot learning performance. Code
and models are available at https://github.com/bdaiinstitute/theia.
Cycle3D: High-quality and Consistent Image-to-3D Generation via
  Generation-Reconstruction Cycle
Authors: name: Zhenyu Tang, name: Junwu Zhang, name: Xinhua Cheng, name: Wangbo Yu, name: Chaoran Feng, name: Yatian Pang, name: Bin Lin, name: Li Yuan
Published: 2024-07-28 17:58:35

Recent 3D large reconstruction models typically employ a two-stage process,
including first generate multi-view images by a multi-view diffusion model, and
then utilize a feed-forward model to reconstruct images to 3D content.However,
multi-view diffusion models often produce low-quality and inconsistent images,
adversely affecting the quality of the final 3D reconstruction. To address this
issue, we propose a unified 3D generation framework called Cycle3D, which
cyclically utilizes a 2D diffusion-based generation module and a feed-forward
3D reconstruction module during the multi-step diffusion process. Concretely,
2D diffusion model is applied for generating high-quality texture, and the
reconstruction model guarantees multi-view consistency.Moreover, 2D diffusion
model can further control the generated content and inject reference-view
information for unseen views, thereby enhancing the diversity and texture
consistency of 3D generation during the denoising process. Extensive
experiments demonstrate the superior ability of our method to create 3D content
with high-quality and consistency compared with state-of-the-art baselines.
VolDoGer: LLM-assisted Datasets for Domain Generalization in
  Vision-Language Tasks
Authors: name: Juhwan Choi, name: Junehyoung Kwon, name: JungMin Yun, name: Seunguk Yu, name: YoungBin Kim
Published: 2024-07-29 08:38:46

Domain generalizability is a crucial aspect of a deep learning model since it
determines the capability of the model to perform well on data from unseen
domains. However, research on the domain generalizability of deep learning
models for vision-language tasks remains limited, primarily because of the lack
of required datasets. To address these challenges, we propose VolDoGer:
Vision-Language Dataset for Domain Generalization, a dedicated dataset designed
for domain generalization that addresses three vision-language tasks: image
captioning, visual question answering, and visual entailment. We constructed
VolDoGer by extending LLM-based data annotation techniques to vision-language
tasks, thereby alleviating the burden of recruiting human annotators. We
evaluated the domain generalizability of various models, ranging from
fine-tuned models to a recent multimodal large language model, through
VolDoGer.
Visual Riddles: a Commonsense and World Knowledge Challenge for Large
  Vision and Language Models
Authors: name: Nitzan Bitton-Guetta, name: Aviv Slobodkin, name: Aviya Maimon, name: Eliya Habba, name: Royi Rassin, name: Yonatan Bitton, name: Idan Szpektor, name: Amir Globerson, name: Yuval Elovici
Published: 2024-07-28 11:56:03

Imagine observing someone scratching their arm; to understand why, additional
context would be necessary. However, spotting a mosquito nearby would
immediately offer a likely explanation for the person's discomfort, thereby
alleviating the need for further information. This example illustrates how
subtle visual cues can challenge our cognitive skills and demonstrates the
complexity of interpreting visual scenarios. To study these skills, we present
Visual Riddles, a benchmark aimed to test vision and language models on visual
riddles requiring commonsense and world knowledge. The benchmark comprises 400
visual riddles, each featuring a unique image created by a variety of
text-to-image models, question, ground-truth answer, textual hint, and
attribution. Human evaluation reveals that existing models lag significantly
behind human performance, which is at 82\% accuracy, with Gemini-Pro-1.5
leading with 40\% accuracy. Our benchmark comes with automatic evaluation tasks
to make assessment scalable. These findings underscore the potential of Visual
Riddles as a valuable resource for enhancing vision and language models'
capabilities in interpreting complex visual scenarios.