Summarize this and list important information as bullet points: Skip to content

Advertisement
Menu

SearchSearch
Search...
Published on May 23, 2023
In Endless Origins
Meta Finally Cracks the LLM Code Without RLHF
So far, Meta has been reluctant with LLMs and chatbots, but is now rushing into the open-source without reinforcement learning with human feedback (RLHF)
By Mohit Pandey
 
 
 
 
Meta Builds a Moat in AI


Listen to this story
Meta is back in the LLM business. Google had claimed that neither they nor does OpenAI have a moat when it comes to AI, but also acknowledged that open source will pave the way for the future of generative AI. The interesting thing about this is Meta is being touted as the leader for open-source generative AI by developers because of its open-source LLM, LLaMa. 

Meta has released another open-source model called MMS: Massively Multilingual Speech. The model is made for speech-to-text and text-to-speech in 1100 languages with the ability to recognise 4,000 spoken languages. French computer scientist Yann LeCun said that the model has half the error rate of OpenAI’s Whisper. This is a huge advancement when it comes to multilingual speech AI technology.


Besides this, yesterday, LeCun shared Meta AI’s latest breakthrough in LLMs. LIMA, made in collaboration with Carnegie Mellon University, University of Southern California, and Tel Aviv University, is a 65 billion parameter model built with LLaMA and fine-tuned with a standard supervised loss with only 1,000 carefully curated prompts and responses. 

The most interesting part is that it does not require RLHF (reinforcement learning with human feedback), which is being implemented within ChatGPT and Bard, and told as the differentiating and the most important factor for its working. There is a possibility that this model might be implementing self-supervised learning, which LeCun has been the biggest proponent of all this while.

When it comes to performance, LeCun claims that this LIMA is on par with GPT-4 and Bard. People have been critical of this. And though excited about the development, the community is also calling out that the benchmarks to compare it with GPT-4 and Bard are not accurate. The paper is still under review.

Recovering Doomer
LeCun said in an interview, “The platform that will win will be an open one. Progress is faster when it is open.” This might seem a little contrary to the stance that the company took during this AI chatbot race. Before releasing LLaMa in February, the company was keeping itself away from LLMs. LeCun used to call these technologies “great but nothing revolutionary” and thus was also sceptical about releasing such technology to the public. 

A lot of it is because the company had actually ventured into the field before any of the competition. BlenderBot and Galactica, two separate chatbots were released by the company but had to be shut down when they turned into hallucinating disasters. Probably, a lot of cynicism towards this LLM-based technology was just trauma from the past. 

Just before the release of LLaMa, Mark Zuckerberg had made a Facebook post talking about reorganising its teams’ structure and focusing on generative AI technology. But, instead of building a chatbot to compete with Microsoft, OpenAI, and Google, the company took an interesting turn, and gave away its most precious asset to open source.

The still-under-review paper of LIMA shows the possibility. There is a chance that the company might venture into the field of chatbots, again. The company had been reluctant to release its models to the public earlier because of their capability of generating misinformation and toxic content, as explained by LeCun.


Now the company has taken a different approach and released its model to the general public that also includes weights, making it a lot easier to make chatbots for developers. Seems like Meta wants a thousand chatbots to exist, but won’t make one.

Is LLaMa the moat for Meta?
The tables have turned. Google and OpenAI, the companies which were in a rush to make their chatbots public, even though keeping the technology behind closed doors for most of the cases, are now considering regulations around the technology. Meanwhile, Meta, which was always not so sure about the technology, is now making their most capable models open source. 

Seems like Meta is banking on the open-source community instead of trying to build a chatbot like Google or OpenAI. For Meta, the community that is building models with LLaMa is the moat. Just like developers and creators flock around StabilityAI for its open source software, people are building model after model over LLaMa, that are outperforming GPT-4 or Bard on less computation requirements as well.

Google’s leaked document said, “The modern internet runs on open source for a reason and we should not be expected to catch up with it.” Even though Google, Microsoft, and OpenAI’s heads were called to the White House to discuss regulations around AI, Meta was left out, citing the reason that the meeting “was focused on companies currently leading the space”. 

Even though Meta does not have a customer-facing AI product, they are not too behind on it as well. According to reports, Zuckerberg, during April’s earnings call, mentioned ‘AI’ at least 27 times, which might be less compared to Sundar Pichai chanting it 140 times at Google I/O, but still the company has been integrating generative AI into a lot of its apps, similar to Google. 

Moreover, the FAIR department of the company has been publishing papers at a rate faster than OpenAI or Google, both of which are now more concerned about protecting their technologies.

It might be too early to tell if Meta taking sides with the open-source community would actually make a mark for them in the AI arms race. But there is no doubt that LLaMa has indeed made an impact on Google and OpenAI. Sam Altman has already discussed imposing restrictions around the technology with the US Senate.

 
 
 
 
Download our Mobile App


Mohit Pandey
Mohit dives deep into the AI world to bring out information in simple, explainable, and sometimes funny words. He also holds a keen interest in photography, filmmaking, and the gaming industry.



Subscribe to our newsletter
Join our editors every weekday evening as they steer you through the most significant news of the day.
Email
Enter your email
Your newsletter subscriptions are subject to AIM Privacy Policy and Terms and Conditions.
Our Upcoming Events
Deep Learning DevCon 2023
May 27, 2023 | Online
Register
Rakuten Product Conference 2023
31st May - 1st Jun '23 | Online
Register
MachineCon 2023 India
Jun 23, 2023 | Bangalore
Register
MachineCon 2023 USA
Jul 21, 2023 | New York
More Details
Cypher 2023
Oct 11-13, 2023 | Bangalore
Register
3 Ways to Join our Community
Telegram group
Discover special offers, top stories, upcoming events, and more.

JOIN TELEGRAM
Discord Server
Stay Connected with a larger ecosystem of data science and ML Professionals

JOIN DISCORD COMMUNITY
Subscribe to our Daily newsletter
Get our daily awesome stories & videos in your inbox
SUBSCRIBE
MOST POPULAR

Council Post: Towards An Ethical Tech Revolution: Building Responsible AI Practices
A responsible AI framework enables companies to track and mitigate bias and create transparent and explainable AI models, prevent misuse and adverse effects of AI, determine who to be held responsible if something goes wrong, and ensure compliance with security, privacy, and associated regulations.


Generative AI in Gaming is a Double-edged Sword
Twelve years after its release, The Elder Scrolls V: Skyrim is being reinvented with the help of generative AI


How Splunk Can Save the Day for Enterprise AI
“The use of AI could raise compliance and privacy concerns, particularly in industries such as finance and healthcare”


Has OpenAI Lost the Open Race?
Will OpenAI go back to its initial roots of being ‘open’ with a proposed plan to release an open-source model?


Google Goes Silent on Quantum Computing Project. What’s Cooking?
Google’s ‘bold and responsible AI’ approach might lead to their fall.


LLM Wars: PaLM 2 VS GPT-4
Google claims that PaLM 2 demonstrates enhanced reasoning capabilities compared to GPT-4, particularly in tasks like WinoGrande and DROP, with slight advantage in ARC-C as well.


How this Company Utilises Generative AI to Transform Online Branding
SingleInterface is currently using LLMs to try to replicate the personalities of a sales agent and solving the problem of customers

Is AI Copyright Really Necessary?
Is AI Copyright Really Necessary? 
Copyright laws and original author attribution sounds like a fair practice unless looked closely.


Council Post: Shaping Tomorrow – The Transformative Potential Of Quantum Machine Learning
By navigating these complexities, we can ensure that Quantum Machine Learning does not just become a part of our future but shapes it, driving us towards a more sustainable and technologically advanced society.


Killed By Amazon (Part 1)
Since the list is too exhaustive, we have made it a two-part series

Our Mission Is To Bring About Better-Informed And More Conscious Decisions About Technology Through Authoritative, Influential, And Trustworthy Journalism.
SHAPE THE FUTURE OF TECH
CONTACT US ⟶

AIM discovers new ideas and breakthroughs that create new relationships, new industries, and new ways of thinking. AIM is the crucial source of knowledge and concepts that make sense of a reality that is always changing.

Our discussions shed light on how technology is transforming many facets of our life, from business to society to culture.
Facebook
Twitter
Youtube
Linkedin
Instagram
WHO WE ARE
ABOUT US
NEWSLETTERS
PODCAST
VIDEOS
EVENTS
CAREERS
EDITORIAL CALENDAR
CONTACT US
OUR BRANDS
MACHINEHACK
BEST FIRM CERTIFICATION
AIM RECRUITS
AIM RESEARCH
AIM LEADERS COUNCIL
ASSOCIATION OF DATA SCIENTISTS
PENETRATION AND MATURITY (PEMA) QUADRANT
OUR SERVICES
ADVERTISE WITH US
BESPOKE EVENTS
EMPLOYER BRANDING
HACKATHONS
RECRUITMENT SERVICES
SYNDICATED RESEARCH
ADVISORY
SKILL ASSESSMENT
OUR CONFERENCES
CYPHER
MACHINECON INDIA
MACHINECON USA
MACHINE LEARNING DEVELOPERS SUMMIT
THE RISING | WOMEN IN TECH CONFERENCE
DATA ENGINEERING SUMMIT
AWARDS
AI100 AWARDS
DATA SCIENCE EXCELLENCE AWARDS
40 UNDER 40 DATA SCIENTISTS
WOMEN IN TECH LEADERSHIP
BEST FIRM FOR WOMEN IN TECH
DATA ENGINEERING AWARDS
FOR DEVELOPERS
HACKATHONS
PRACTICE SKILLS
COURSES
DISCUSSION BOARD
JOB BOARD
TECHNICAL BLOGS
AIM CAMPUS AMBASSADOR
© ANALYTICS INDIA MAGAZINE PVT LTD & AIM MEDIA HOUSE LLC 2023

Terms of use
Privacy Policy
Copyright