Bhadresh Savani's picture

Bhadresh Savani

bhadresh-savani

·

https://www.linkedin.com/in/bhadreshsavani/

AI & ML interests

NLP, Deep Learning, ML

Recent Activity

Reacted to lin-tan's post with 🔥 about 5 hours ago

Can language models replace developers? #RepoCod says “Not Yet”, because GPT-4o and other LLMs have <30% accuracy/pass@1 on real-world code generation tasks. - Leaderboard https://lt-asset.github.io/REPOCOD/ - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD @jiang719 @shanchao @Yiran-Hu1007 Compared to #SWEBench, RepoCod tasks are - General code generation tasks, while SWE-Bench tasks resolve pull requests from GitHub issues. - With 2.6X more tests per task (313.5 compared to SWE-Bench’s 120.8). Compared to #HumanEval, #MBPP, #CoderEval, and #ClassEval, RepoCod has 980 instances from 11 Python projects, with - Whole function generation - Repository-level context - Validation with test cases, and - Real-world complex tasks: longest average canonical solution length (331.6 tokens) and the highest average cyclomatic complexity (9.00) Introducing hashtag #RepoCod-Lite 🐟 for faster evaluations: 200 of the toughest tasks from RepoCod with: - 67 repository-level, 67 file-level, and 66 self-contains tasks - Detailed problem descriptions (967 tokens) and long canonical solutions (918 tokens) - GPT-4o and other LLMs have < 10% accuracy/pass@1 on RepoCod-Lite tasks. - Dataset: https://huggingface.co/datasets/lt-asset/REPOCOD_Lite #LLM4code #LLM #CodeGeneration #Security

View all activity

Organizations

bhadresh-savani's activity

upvoted 2 articles 3 months ago

Article

Llama-3.1-Storm-8B: Improved SLM with Self-Curation + Model Merging

By

•

Aug 19

• 73

Article

Train a Llama model from scratch

By

•

Jul 29

• 45

upvoted 5 articles 4 months ago

Article

Google releases Gemma 2 2B, ShieldGemma and Gemma Scope

Jul 31

• 59

Article

Serverless Inference with Hugging Face and NVIDIA NIMs

Jul 29

• 28

Article

Google Cloud TPUs made available to Hugging Face users

Jul 9

• 19

Article

Our Transformers Code Agent beats the GAIA benchmark!

Jul 1

• 46

Article

Llama 3.1 - 405B, 70B & 8B with multilinguality and long context

Jul 23

• 215

upvoted a collection 4 months ago

Llama 3.1

This collection hosts the transformers and original repos of the Llama 3.1, Llama Guard 3 and Prompt Guard models • 11 items • Updated Sep 25 • 624

upvoted a collection 6 months ago

Granite Code Models

A series of code models trained by IBM licensed under Apache 2.0 license. We release both the base pretrained and instruct models. • 23 items • Updated 19 days ago • 178

upvoted a collection 9 months ago

Gemma release

Groups the Gemma models released by the Google team. • 40 items • Updated Jul 31 • 325