Danielus's picture

Danielus

danielus

·

DanielusG

AI & ML interests

None yet

Recent Activity

liked a model about 1 hour ago

deepseek-ai/DeepSeek-V3-Base

reacted to YerbaPage's post with 👀 8 days ago

Curated list of **Repository-level Code Generation** papers & benchmarks! 🔥 Stay ahead with the latest in: ✅ Repo-level Issue Resolution ✅ Repo-level Code Completion ✅ Datasets & Benchmarks 👉 Check it out: https://github.com/YerbaPage/Awesome-Repo-Level-Code-Generation 🔥

reacted to m-ric's post with 🔥 11 days ago

𝗣𝗼𝘁𝗲𝗻𝘁𝗶𝗮𝗹 𝗽𝗮𝗿𝗮𝗱𝗶𝗴𝗺 𝘀𝗵𝗶𝗳𝘁 𝗶𝗻 𝗟𝗟𝗠𝘀: 𝗻𝗲𝘄 𝗽𝗮𝗽𝗲𝗿 𝗯𝘆 𝗠𝗲𝘁𝗮 𝗰𝗹𝗮𝗶𝗺𝘀 𝘁𝗵𝗮𝘁 𝘄𝗲 𝗰𝗮𝗻 𝗴𝗲𝘁 𝗿𝗶𝗱 𝗼𝗳 𝘁𝗼𝗸𝗲𝗻𝗶𝘇𝗲𝗿𝘀! 🥳 Current LLMs process text by first splitting it into tokens. They use a module named "tokenizer", that -spl-it-s- th-e- te-xt- in-to- arbitrary tokens depending on a fixed dictionnary. On the Hub you can find this dictionary in a model's files under tokenizer.json. ➡️ This process is called BPE tokenization. It is suboptimal, everyone says it. It breaks text into predefined chunks that often fail to capture the nuance of language. But it has been a necessary evil in language models since their inception. 💥 In Byte Latent Transformer (BLT), Meta researchers propose an elegant solution by eliminating tokenization entirely, working directly with raw bytes while maintaining efficiency through dynamic "patches." This had been tried before with different byte-level tokenizations, but it's the first time that an architecture of this type scales as well as BPE tokenization. And it could mean a real paradigm shift! 👏👏 🏗️ 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲: Instead of a lightweight tokenizer, BLT has a lightweight encoder that process raw bytes into patches. Then the patches are processed by the main heavy-duty transformers as we do normally (but for patches of bytes instead of tokens), before converting back to bytes. 🧩 𝗗𝘆𝗻𝗮𝗺𝗶𝗰 𝗣𝗮𝘁𝗰𝗵𝗶𝗻𝗴: Instead of fixed tokens, BLT groups bytes based on their predictability (measured by entropy) - using more compute for complex sequences and efficiently handling simple ones. This allows efficient processing while maintaining byte-level understanding. I hope this breakthrough is confirmed and we can get rid of all the tokenizer stuff, it will make model handling easier! Read their paper here 👉 https://dl.fbaipublicfiles.com/blt/BLT__Patches_Scale_Better_Than_Tokens.pdf

View all activity

Organizations

None yet

danielus's activity

upvoted 2 collections 6 months ago

Gemma 2 Release

15 items • Updated 13 days ago • 206

DeepSeekCoder-V2

6 items • Updated Sep 5 • 81

upvoted an article 7 months ago

Article

Orchestration of Experts: The First-Principle Multi-Model System

By

•

May 30

• 15

upvoted a collection 7 months ago

C4AI Aya 23

Aya 23 is an open weights research release of an instruction fine-tuned model with highly advanced multilingual capabilities. • 4 items • Updated 23 days ago • 50

upvoted an article 7 months ago

Article

2024-04-22 - Hub Incident Post Mortem

By

•

May 17

• 17

upvoted an article 8 months ago

Article

How to Finetune phi-3 on MacBook Pro

By

•

Apr 24

• 64

upvoted a collection 8 months ago

LLM ITA

Open-Source Language Models Finetuned for Italian • 4 items • Updated Oct 19 • 5

upvoted an article 8 months ago

Article

Welcome Llama 3 - Meta's new open LLM

Apr 18

• 281

upvoted 2 papers 10 months ago

NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models

Paper • 2403.03100 • Published Mar 5 • 34

A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts

Paper • 2402.09727 • Published Feb 15 • 36