Hugging Face's logo
Merve Noyan's picture
509 118 163

Merve Noyan PRO

merve
cameronarbis's profile picture VedChen's profile picture barisd's profile picture
·
mervenoyann
merveenoyan

AI & ML interests

VLMs, vision & co

Articles

Organizations

Hugging Face's profile picture Google's profile picture PyTorch Image Models's profile picture Templates's profile picture Keras's profile picture tensorflow's profile picture Mukayese's profile picture HugGAN Community's profile picture Hugging Face Fellows's profile picture HuggingFaceM4's profile picture HF Canonical Model Maintainers's profile picture scikit-learn's profile picture fastai X Hugging Face Group 2022's profile picture Huggingface Projects's profile picture Kornia AI's profile picture skops-tests's profile picture Turkish T5 - BERT - GPT-2's profile picture SODA's profile picture 2023 Jan Offsite hackathon's profile picture Keras Dreambooth Event's profile picture Deprem Yapay Zeka's profile picture Deprem Private's profile picture Turkish NLP Dataset Creators's profile picture Demo Crafters 🤗 's profile picture Huggingface.js's profile picture scikit-learn's profile picture JAX ♥️ Diffusers 🧨's profile picture boun-tabi-LMG's profile picture Notebooks-explorers's profile picture Blog-explorers's profile picture Hugging Face for Computer Vision's profile picture Hacktoberfest 2023's profile picture adept-hf-collab's profile picture Niansuh AI's profile picture ZeroGPU Explorers's profile picture Llava Hugging Face's profile picture MLX Community's profile picture Social Post Explorers's profile picture kotol's profile picture Top Contributors: Profile Followers's profile picture Dev Mode Explorers's profile picture Paris AI Running Club's profile picture CVPR2024's profile picture

Posts 31

view post
Post
1553
Do we fully leverage ViT encoders in vision language models?

A new paper (by @HuanjinYao et al) built a dense connector that does it better! HuanjinYao/DenseConnector-v1.5-8B
HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea

VLMs consist of an image encoder block, a projection layer that projects image embeddings to text embedding space and then a text decoder sequentially connected 📖
This paper explores using intermediate states of image encoder and not a single output 🤩
The authors explore three different ways of instantiating dense connector: sparse token integration, sparse channel integration and dense channel integration. (see paper on how they do it Dense Connector for MLLMs (2405.13800))

They explore all three of them integrated to LLaVA 1.5 and found out each of the new models are superior to the original LLaVA 1.5 🥹 I tried the model and it seems to work very well. As part of the release, the authors have released various ckpts based on different decoders (Vicuna 7/13B and Llama 3-8B) that you can find in the collection 🤗

view post
Post
952
We will be providing ZeroGPU grants (for Spaces inference) to those who want to fine-tune PaliGemma and build a Space 🔥

You can pick any dataset of your choice!

Example code: https://colab.research.google.com/drive/1x_OEphRK0H97DqqxEyiMewqsTiLD_Xmi?usp=sharing (you can use a lower GPU with QLoRA)

Datasets:
https://huggingface.co/datasets?task_categories=task_categories:text-to-image&sort=trending
https://huggingface.co/datasets?task_categories=task_categories:image-to-text&sort=trending
View all posts

Collections 21