Faiss - Vector Compression with PQ and IVFPQ (in Python)

So far we’ve worked through the logic behind a simple, readable implementation of product quantization (PQ) in Python for semantic search. Realistically we wouldn’t use this because it is not optimized and we already have excellent implementations elsewhere. Instead, we would use a library like Faiss (Facebook AI Similarity Search) — or a production-ready service like Pinecone.

We’ll take a look at how we can build a PQ index in Faiss, and we’ll even take a look at combining PQ with an Inverted File (IVF) step to improve search speed.

Before we start, we need to get data. We will be using the Sift1M dataset. It can be downloaded and opened using this script:
https://gist.github.com/jamescalam/928a374b85daffa49a565f3dc18d059c#file-get_sift1m-ipynb

🌲 Pinecone article:
https://www.pinecone.io/learn/product-quantization/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Product Quantization for Vector Similarity Search (+ Python)

Vector similarity search can require huge amounts of memory. Indexes containing 1M dense vectors (a small dataset in today’s world) will often require several GBs of memory to store. When building recommendation systems or semantic search engines, this is not acceptable.

The problem of excessive memory usage is exasperated by high-dimensional data, and with ever-increasing dataset sizes, this can very quickly become unmanageable.

Product quantization (PQ) is a popular method for dramatically compressing high-dimensional vectors to use 97% less memory, and for making nearest-neighbor search speeds 5.5x faster in our tests.

A composite IVF+PQ index speeds up the search by another 16.5x without affecting accuracy, for a whopping total speed increase of 92x compared to non-quantized indexes.

🌲 Pinecone article:
https://www.pinecone.io/learn/product-quantization/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Composite Indexes and the Faiss Index Factory

In the world of vector search, there are many indexing methods and vector processing techniques that allow us to prioritize between recall, latency, and memory usage.

Using specific methods such as IVF, PQ, or HNSW, we can often return good results. But for best performance we will usually want to use composite indexes.

We can view a composite index as a step-by-step process of vector transformations and one or more indexing methods. Allowing us to place multiple indexes and/or processing steps together to create our ‘ideal’ index.

For example, we can use an inverted file (IVF) index to reduce the scope of our search (increasing search speed), and then add a compression technique such as product quantization (PQ) to keep larger indexes within a reasonable size limit.

Where there is the ability to customize indexes, there is the risk of producing indexes with unnecessarily poor recall, latency, or memory usage.

We must know how composite indexes work if we want to build robust and high-performance vector similarity search applications. It is essential to understand where different indexes or vector transformations can be used — and when they are not needed.

Part 2: https://youtu.be/3Wqh4iUupbM

🌲 Pinecone article:
https://www.pinecone.io/learn/composite-indexes/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:54 Composite Indexes
06:43 Faiss Index Factory
11:34 Why we use Index Factory
17:11 Outro


Best Indexes for Similarity Search in Faiss

In the world of vector search, there are many indexing methods and vector processing techniques that allow us to prioritize between recall, latency, and memory usage.

Using specific methods such as IVF, PQ, or HNSW, we can often return good results. But for best performance we will usually want to use composite indexes.

We can view a composite index as a step-by-step process of vector transformations and one or more indexing methods. Allowing us to place multiple indexes and/or processing steps together to create our ‘ideal’ index.

For example, we can use an inverted file (IVF) index to reduce the scope of our search (increasing search speed), and then add a compression technique such as product quantization (PQ) to keep larger indexes within a reasonable size limit.

Where there is the ability to customize indexes, there is the risk of producing indexes with unnecessarily poor recall, latency, or memory usage.

We must know how composite indexes work if we want to build robust and high-performance vector similarity search applications. It is essential to understand where different indexes or vector transformations can be used — and when they are not needed.

Part 1: https://youtu.be/GEhmmcx1lvM

🌲 Pinecone article:
https://www.pinecone.io/learn/composite-indexes/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:30 IVFADC
03:30 IVFADC in Faiss
07:29 Multi-D-ADC
09:17 Multi-D-ADC in Faiss
14:43 IVF-HNSW
21:39 IVF-HNSW in Faiss
25:58 Outro


How to Build a Bert WordPiece Tokenizer in Python and HuggingFace

Building a transformer model from scratch can often be the only option for many more specific use cases. Although BERT and other transformer models have been pre-trained for a vast number of languages and domains, they do not cover everything.

Often, it is these less common use cases that stand to gain the most from having someone come along and build a specific transformer model. It could be for an uncommon language or less tech-savvy domain.

BERT is the most popular transformer for a wide range of language-based machine learning - from sentiment analysis to question and answering, BERT has enabled a diverse range of innovation across many borders and industries.

The first step for many in designing a new BERT model is the tokenizer. In this article, we'll take a look at the WordPiece tokenizer used by BERT - and see how we can build our own from scratch.

📕 Medium article:
https://towardsdatascience.com/how-to-build-a-wordpiece-tokenizer-for-bert-f505d97dddbb

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free Article link (if you don't have Medium membership):
https://towardsdatascience.com/how-to-build-a-wordpiece-tokenizer-for-bert-f505d97dddbb?sk=eea06e01c9faecd939e10589e9de1291


Metadata Filtering for Vector Search + Latest Filter Tech

Vector similarity search makes massive datasets searchable in fractions of a second. Yet despite the brilliance and utility of this technology, often what seem to be the most straightforward problems are the most difficult to solve. Such as filtering.

Filtering takes the top place in being seemingly simple — but actually incredibly complex. Applying fast-but-accurate filters when performing a vector search (ie, nearest-neighbor search) on massive datasets is a surprisingly stubborn problem.

This article explains the two common methods for adding filters to vector search, and their serious limitations. Then we will explore Pinecone’s solution to filtering in vector search.

📣 Get the API key!
https://www.pinecone.io/start/

🌲 Pinecone article:
https://www.pinecone.io/learn/vector-search-filtering/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:24 Vector Search Recap
02:03 Why Filter?
02:56 Metadata Filtering 101
07:48 Pre-filtering
09:37 Post-filtering
11:30 Single-Stage Filtering
12:22 Vectors and Metadata Code
13:58 Connecting to Pinecone
14:55 Building Query Vector
16:47 Querying
21:37 First Filter
24:40 Adding More Conditions
27:03 Filtering with Numbers
30:55 Search Speed and Filtering
33:44 Outro


Build NLP Pipelines with HuggingFace Datasets

HF Datasets is an essential tool for NLP practitioners - hosting over 1.4K (mostly) high-quality language-focused datasets, and an easy-to-use treasure trove of functions for building efficient pre-processing pipelines.

In this article, we will take a look at the massive repository of datasets available, and explore some of the library's brilliant data processing capabilities.

📕 Medium article:
https://towardsdatascience.com/build-nlp-pipelines-with-huggingface-datasets-d597ff5f68ad

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

📖 Free Article Access (if you don't have Medium membership!):
https://towardsdatascience.com/build-nlp-pipelines-with-huggingface-datasets-d597ff5f68ad?sk=948106e47e64bc3e9e8a1358b0568d48


HNSW for Vector Search Explained and Implemented with Faiss (Python)

Hierarchical Navigable Small World (HNSW) graphs are among the top-performing indexes for vector similarity search. HNSW is a hugely popular technology that time and time again produces state-of-the-art performance with super-fast search speeds and flawless recall — HNSW is not to be missed.

Despite being a popular and robust algorithm for approximate nearest neighbors (ANN) searches, understanding how it works is far from easy.

This video helps demystify HNSW and explains this intelligent algorithm in an easy-to-understand way. Towards the end of the video, we'll look at how to implement HNSW using Faiss and which parameter settings give us the performance we need.

🌲 Pinecone article:
https://www.pinecone.io/learn/hnsw/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:41 Foundations of HNSW
08:41 How HNSW Works
16:38 The Basics of HNSW in Faiss
21:40 How Faiss Builds an HNSW Graph
26.49 Building the Best HNSW Index
33:33 Fine-tuning HNSW
34:30 Outro


Intro to APIs in Python - API Series #1

Taking those first steps into interacting with the web using Python can seem daunting - but it need not be. It is a surprisingly simple process, with well established rules and guidelines.

We'll cover the absolute essentials for getting started, including:

- Application Program Interfaces (APIs)
- Javascript Object Notation (JSON)
- Requests with Python
- Real world use-cases

📕 Article:
https://towardsdatascience.com/quick-fire-guide-to-apis-in-python-891dd98c8877

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

📖 Free Access Link (if you don't have Medium membership):
https://towardsdatascience.com/quick-fire-guide-to-apis-in-python-891dd98c8877?sk=7c159ba45154db23abcc6a7f9de4f910

Geocoding Docs:
https://developers.google.com/maps/documentation/geocoding/cloud-setup

GitHub Docs:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

00:00 Intro
00:20 What is an API?
01:47 RESTful APIs
05:26 API Methods
07:20 HTTP Codes (200s)
08:14 HTTP Codes (400s)
10:00 JSON Format
11:21 Talking to APIs in Python
14:30 Google Geocoding API
22:08 GitHub API
27:48 Outro


Intro to Dense Vectors for NLP and Vision

There is perhaps no greater component to the success of modern Natural Language Processing (NLP) technology than vector representations of language. The meteoric early 2010s rise of NLP was ignited with the introduction of word2vec by a team lead by Tomáš Mikolov in 2013.

Word2vec is one of the most iconic and earliest examples of dense vectors representing text. But since the days of word2vec, developments in representing language have advanced at ludicrous speeds.

This video will explore *why* we use dense vectors — and some of the best approaches to building dense vectors available today.

🌲 Pinecone article:
https://www.pinecone.io/learn/dense-vector-embeddings-nlp/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:50 Why Dense Vectors?
03:55 Word2vec and Representing Meaning
08:40 Sentence Transformers
09:58 Sentence Transformers in Python
15:08 Question-Answering
18:18 DPR in Python
29:55 Vision Transformers
33:22 OpenAI's CLIP in Python
42:49 Review and What's Next


API Series #2 - Building an API with Flask in Python

Next video - how to deploy to the cloud: https://youtu.be/3fsIcMgUOY8

How can we set up a way to communicate from one software instance to another? It sounds simple, and — to be completely honest — it is.

All we need is an API.

An API (Application Programming Interface) is a simple interface that defines the types of requests (demands/questions, etc.) that can be made, how they are made, and how they are processed.

In our case, we will be building an API that allows us to send a range of GET/POST/PUT/PATCH/DELETE requests (more on this later), to different endpoints, and return or modify data connected to our API.

We will be using the Flask framework to create our API and Insomnia to test it.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🕹 Medium article:
https://towardsdatascience.com/the-right-way-to-build-an-api-with-python-cd08ab285f8f

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

Free article link:
https://towardsdatascience.com/the-right-way-to-build-an-api-with-python-cd08ab285f8f?sk=6e2dda4c8b6012767114e12ff34b1464

Download Insomnia:
https://insomnia.rest/download


Intro to Sentence Embeddings with Transformers

Transformers have wholly rebuilt the landscape of natural language processing (NLP). Before transformers, we had okay translation and language classification thanks to recurrent neural nets (RNNs) — their language comprehension was limited and led to many minor mistakes, and coherence over larger chunks of text was practically impossible.

Since the introduction of the first transformer model in the 2017 paper ‘Attention is all you need’, NLP has moved from RNNs to models like BERT and GPT. These new models can answer questions, write articles (maybe GPT-3 wrote this), enable incredibly intuitive semantic search — and much more.

In this video, we will explore how these embeddings have been adapted and applied to a range of semantic similarity applications by using a new breed of transformers called ‘sentence transformers’.

🌲 Pinecone article:
https://www.pinecone.io/learn/sentence-embeddings/

Vectors in ML:
https://www.youtube.com/playlist?list=PLIUOU7oqGTLgz-BI8bNMVGwQxIMuQddJO

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Fine-tune Sentence Transformers the OG Way (with NLI Softmax loss)

Sentence embeddings with transformers can be used across a range of applications, such as semantic textual similarity (STS), semantic clustering, or information retrieval (IR) using concepts rather than words.

This video dives deeper into the training process of the first sentence transformer, sentence-BERT, or more commonly known as SBERT. We will explore the Natural Language Inference (NLI) training approach of softmax loss to fine-tune models for producing sentence embeddings.

Be aware that softmax loss is no longer the preferred approach to training sentence transformers and has been superseded by other methods such as MSE margin and multiple negatives ranking loss. But we’re covering this training method as an important milestone in the development of ever-improving sentence embeddings.

🌲 Pinecone article:
https://www.pinecone.io/learn/train-sentence-transformers-softmax/

Check out the Sentence Transformers library:
https://github.com/UKPLab/sentence-transformers

Talk by Nils Reimers (one of the SBERT creators) on training:
https://www.youtube.com/watch?v=RHXZKUr8qOY

He does more NLP vids too:
https://www.youtube.com/channel/UC1zCuTrfpjT6Sv2kJk-JkvA

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:42 NLI Fine-tuning
01:44 Softmax Loss Training Overview
05:47 Preprocessing NLI Data
12:48 PyTorch Process
19:48 Using Sentence-Transformers
30:45 Results
35:49 Outro


Fine-tune High Performance Sentence Transformers (with Multiple Negatives Ranking)

Transformer-produced sentence embeddings have come a long way in a very short time. Starting with the slow but accurate similarity prediction of BERT cross-encoders, the world of sentence embeddings was ignited with the introduction of SBERT in 2019. Since then, many more sentence transformers have been introduced. These models quickly made the original SBERT obsolete.

How did these newer sentence transformers manage to outperform SBERT so quickly? The answer is multiple negatives ranking (MNR) loss.

This video will cover what MNR loss is, the data it requires, and how to implement it to fine-tune our own high-quality sentence transformers.

Implementation will cover two approaches. The first is more involved, and outlines the exact steps to fine-tune the model (we'll just run over it quickly). The second approach makes use of the sentence-transformers library’s excellent utilities for fine-tuning.

🌲 Pinecone article:
https://www.pinecone.io/learn/fine-tune-sentence-transformers-mnr/

Check out the Sentence Transformers library:
https://github.com/UKPLab/sentence-transformers

Talk by Nils Reimers (one of the SBERT creators) on training:
https://www.youtube.com/watch?v=RHXZKUr8qOY

He does more NLP vids too:
https://www.youtube.com/channel/UC1zCuTrfpjT6Sv2kJk-JkvA

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:02 NLI Training Data
02:56 Preprocessing
10:11 SBERT Finetuning Visuals
14:14 MNR Loss Visual
16:37 MNR in PyTorch
23:04 MNR in Sentence Transformers
34:20 Results
36:14 Outro


Hybrid Search Walkthrough in Pinecone

Pinecone offers a production-ready vector database for high performance and reliable *semantic search* at scale. But did you know Pinecone's semantic search can be paired with the more traditional keyword search?

Semantic search is a compelling technology allowing us to search using abstract concepts and *meaning* rather than relying on specific words. However, sometimes a simple keyword search can be just as valuable — especially if we know the exact wording of what we're searching for.

In this video, we will explore these features through a start-to-finish example of basic keyword search in Pinecone.

🌲 Check the docs:
https://www.pinecone.io/docs/examples/basic-hybrid-search/

🔑 Free API key:
https://app.pinecone.io

00:52 How Hybrid Search Works
01:25 Preprocessing
03:01 Creating Keywords
05:34 Creating an Index
06:50 Data Upsert
08:33 Query Setup
10:52 Keyword Search
12:31 OR Logic
14:49 AND Logic
15:10 Negation
17:04 Outro


API Series #3 - How to Deploy Flask APIs to the Cloud (GCP)

Building that first API is for many of us, a significant step towards creating impactful tools that may one day be used by many developers. But often those APIs don't make it out of our local machines.

Fortunately, it's incredibly easy to deploy APIs. Assuming you have no idea what you're doing right now - you will probably be deploying your first API in around ten minutes.

I'm not joking, it's super easy. Let's get started.

📕 Article:
https://towardsdatascience.com/how-to-deploy-a-flask-api-8d54dd8d8b8a

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

📖 Free article link:
TO ADD


All You Need to Know on Multilingual Sentence Vectors (1 Model, 50+ Languages)

We’ve learned about how sentence transformers can be used to create high-quality vector representations of text. We can then use these vectors to find similar vectors, which can be used for many applications such as semantic search or topic modeling.

These models are very good at producing meaningful, information-dense vectors. But they don’t allow us to compare sentences across different languages.

Often this may not be a problem. However, the world is becoming increasingly interconnected, and many companies span across multiple borders and languages. Naturally, there is a need for sentence vectors that are language agnostic.

Unfortunately, very few textual similarity datasets span multiple languages, particularly for less common languages. And the standard training methods used for sentence transformers would require these types of datasets.

Different approaches need to be used. Fortunately, some techniques allow us to extend models to other languages using more easily obtained language translations.

In this video, we will cover how multilingual models work and are built. We’ll learn how to develop our own multilingual sentence transformers, the datasets to look for, and how to use high-performing pretrained multilingual models.

🌲 Pinecone article:
https://www.pinecone.io/learn/multilingual-transformers/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:19 Multilingual Vectors
05:55 Multi-task Training (mUSE)
09:36 Multilingual Knowledge Distillation
11:13 Knowledge Distillation Training
13:43 Visual Walkthrough
14:53 Parallel Data Prep
20:23 Choosing a Student Model
24:55 Initializing the Models
30:05 ParallelSentencesDataset
33:54 Loss and Fine-tuning
36:59 Model Evaluation
39:23 Outro


Question-Answering in NLP (Extractive QA and Abstractive QA)

Search is a crucial functionality in many applications and companies globally. Whether in manufacturing, finance, healthcare, or *almost* any other industry, organizations have vast internal information and document repositories.

Unfortunately, the scale of many companies’ data means that the organization and accessibility of information can become incredibly inefficient. The problem is exacerbated for language-based information. Language is a tool for people to communicate often abstract ideas and concepts. Naturally, ideas and concepts are harder for a computer to comprehend and store in a meaningful way.

How do we minimize this problem? The answer lies with *semantic search*, specifically with the question-answering (QA) flavor of semantic search.

This article will introduce the different forms of QA, the components of these 'QA stacks', and where we might use them.

🌲 Pinecone article:
https://www.pinecone.io/learn/question-answering/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Meaningful Search
01:23 Use-case
02:22 Open Domain QA (ODQA)
06:41 SQuAD Format
10:45 Quick Preprocessing
15:18 Creating Context Vectors Database
23:24 Open-book Extractive QA
32:50 Open-book Abstractive QA
41:53 Closed-book Abstractive QA
47:27 Final Thoughts


Today Unsupervised Sentence Transformers, Tomorrow Skynet (how TSDAE works)

To adapt a pretrained transformer to produce meaningful sentence vectors, we typically need a more supervised fine-tuning approach. We can use datasets like natural language inference (NLI) pairs, labeled semantic textual similarity (STS) data, or parallel data (pairs of translations).

For some domains and languages, such as finance and English, this data is fairly easy to find or gather. But many domains and many languages have very little labeled data. If you can find semantic similarity pairs for the agriculture industry, please let me know. There are many languages, such as Dhivehi, where unlabelled data is hard to find and labelled data practically non-existent.

This means you either spend a very long time gathering tens of thousands of labeled samples or you can try an unsupervised fine-tuning approach.

Unsupervised training methods for sentence transformers are not as effective as their supervised counterparts, but they do work. And if you have no other choice, why not?

In this video, we will introduce the concept of unsupervised fine-tuning for sentence transformers. We will learn to train these models using the unsupervised Transformer-based Sequential Denoising Auto-Encoder (TSDAE) approach.

🌲 Pinecone article:
https://www.pinecone.io/learn/unsupervised-training-sentence-transformers/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Why Language Embedding Matters
05:12 Supervised Methods
05:29 Natural Language Inference
07:15 Semantic Textual Similarity
07:43 Multilingual Training
10:00 TSDAE (Unsupervised)
18:50 Data Preparation
29:05 Initialize Model
32:39 Model Training
36:25 NLTK Error
37:15 Evaluation
41:01 TSDAE vs Supervised Methods
42:42 Why TSDAE is Cool


Making The Most of Data: Augmented SBERT

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

ML models are data-hungry. They consume massive amounts of data to identify generalized patterns and apply those learned patterns to new data.

As models get bigger, so do datasets. And although we have seen an explosion of data in the past decade, it is often not accessible or in an ML-friendly format, especially in niche domains.

For many niche, low-resource domains, finding or annotating a substantial dataset manually is practically impossible.

Fortunately, we don't need to label (or even find) this new data. Instead, we can automatically generate or label data using one or more *data augmentation* techniques.

In this video, we will introduce data augmentation and its application to the field of NLP. We will focus on the 'in-domain' flavor of a particular data-augmentation strategy named augmented SBERT (AugSBERT).

🌲 Pinecone article:
https://www.pinecone.io/learn/data-augmentation/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Building Transformer Tokenizers (Dhivehi NLP #1)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Get in touch with Ashraq:
https://www.linkedin.com/in/ismailashraq/

The language of Dhivehi (or Maldivian) is fascinating. It uses a complex writing system known as Thaana, and I absolutely cannot comprehend any of it. It is so wildly different from anything I know - but, like the archipelago, it looks wonderful.

Ashraq described the difficulty of applying NLP to his native tongue of Dhivehi. There are several reasons for this, which we will explore in this video, and learn how to build an effective Dhivehi WordPiece tokenizer.

📕 Article:
https://towardsdatascience.com/designing-tokenizers-for-low-resource-languages-7faa4ab30ef4

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Article Friend Link (Free Access):
https://towardsdatascience.com/designing-tokenizers-for-low-resource-languages-7faa4ab30ef4?sk=c0c16de9eea7dbe1d2a9c106abf38e1a

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:06 Dhivehi Project
02:28 Hurdles for Low Resource Domains
04:21 Dhivehi Dataset
04:52 Download Dhivehi Corpus
08:25 Tokenizer Components
08:44 Normalizer Component
11:55 Pre-tokenization Component
14:59 Post-tokenization Component
16:26 Decoder Component
17:41 Tokenizer Implementation
21:04 Tokenizer Training
24:22 Post-processing Implementation
27:12 Decoder Implementation
28:07 Saving for Transformers
30:33 Tokenizer Test and Usage
31:36 Download Dhivehi Models
32:21 First Steps


AugSBERT: Domain Transfer for Sentence Transformers

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

When building language models, we can spend months optimizing training and model parameters, but it’s useless if we don't have the correct data.

The success of our language models relies first and foremost on data. The augmented SBERT training strategy can help us.

Given this scenario, we can transfer information from an out-of-domain (or *source*) dataset to our target domain. We will learn how to do this here. First, we will learn to assess which source datasets align best with our target domain quickly. Then we will explain and work through the AugSBERT domain-transfer training strategy.

🌲 Pinecone article:
https://www.pinecone.io/learn/augsbert-domain-transfer/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🔗 n-gram Similarity Script: https://gist.github.com/jamescalam/b73f37017ae32bd6094747c4b0fca94a
🔗 AugSBERT In-Domain Article: https://www.pinecone.io/learn/data-augmentation/

00:00 Why Use Domain Transfer
04:08 Strategy Outline
06:05 Train Source Cross-Encoder
12:44 Cross-Encoder Outcome
15:12 Labeling Target Data
20:31 Training Bi-encoder
23:58 Evaluator Bi-encoder Performance
28:08 Final Points


How to build a Q&A AI in Python (Open-domain Question-Answering)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

How can we design these natural, human-like Q&A interfaces? The answer is open-domain question-answering (ODQA). ODQA allows us to use natural language to query a database.

That means that, given a dataset like a set of internal company documents, online documentation, or as is the case with Google, everything on the world’s internet, we can retrieve relevant information in a natural, more human way.

🌲 Pinecone article:
https://www.pinecone.io/learn/retriever-models/

🔗 Nils YT Talk: https://youtu.be/XNJThigyvos?t=118
🔗 MNR Loss Article:
🔗 Free Pinecone API Key: https://app.pinecone.io/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Why QA
04:05 Open Domain QA
08:24 Do we need to fine-tune?
11:44 How Retriever Training Works
12:59 SQuAD Training Data
16:29 Retriever Fine-tuning
19:32 IR Evaluation
25:58 Vector Database Setup
33:42 Querying
37:41 Final Notes


How to build a Q&A Reader Model in Python (Open-domain QA)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Open-domain question-answering (ODQA) is a wildly popular *pipeline* of databases and language models that allow us to ask a machine human-like questions and return comprehensible and even intelligent answers.

Despite the outward guise of simplicity, ODQA requires a reasonably advanced set of components placed together to enable the *extractive* Q&A functionality.

We call this *extractive* Q&A because the models are not generating an answer. Instead, the answer already exists but is hidden somewhere within potentially thousands, millions, or even more data sources.

By enabling extractive Q&A, we enable a more *intelligent* and *efficient* way to retrieve information from what can be massive stores of data.

🌲 Pinecone article:
https://www.pinecone.io/learn/reader-models/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:13 ODQA Components
03:09 Data Preprocessing
22:35 Fine-tuning


Streamlit for ML #1 - Installation and API

▶ Streamlit for ML Part 2:
https://www.youtube.com/watch?v=U0EoaFFGyTg&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=2

Streamlit has proven itself as an incredibly popular tool for quickly putting together high-quality ML-oriented web apps. More recently, it has seen wider adoption in production environments by ever-larger organizations.

All of this means that there is no better time to pick up some experience with Streamlit. Fortunately, the basics of Streamlit are incredibly easy to learn, and for most tools, this will be more than you need!

In this series, we will introduce Streamlit by building a general knowledge Q&A interface. We will learn about key Streamlit components like write, text_input, container. How to use external libraries like Bootstrap to quickly create new app components. And use caching to speed up our app.

📕 Article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec?sk=ac5e0b7c39938f52162862411a66a58b

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:39 App Outline
03:36 Streamlit Installation
06:15 Streamlit API Basics


Streamlit for ML #2 - ML Models and APIs

▶ Streamlit for ML Part 3:
https://www.youtube.com/watch?v=lYDiSCDcxmc&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=3

Streamlit has proven itself as an incredibly popular tool for quickly putting together high-quality ML-oriented web apps. More recently, it has seen wider adoption in production environments by ever-larger organizations.

All of this means that there is no better time to pick up some experience with Streamlit. Fortunately, the basics of Streamlit are incredibly easy to learn, and for most tools, this will be more than you need!

In this series, we will introduce Streamlit by building a general knowledge Q&A interface. We will learn about key Streamlit components like write, text_input, container. How to use external libraries like Bootstrap to quickly create new app components. And use caching to speed up our app.

🔗 Code to Create Index:
https://gist.github.com/jamescalam/2123ce0bb8a871f48a151a023a7ece67

📕 Article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec?sk=ac5e0b7c39938f52162862411a66a58b

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:47 Creating the Vector DB
08:56 Implementing Retrieval


Streamlit for ML #3 - Make Apps Fast with Caching

▶ Streamlit for ML Part 4:
https://www.youtube.com/watch?v=XdxeKiY2UXg&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=4

Streamlit has proven itself as an incredibly popular tool for quickly putting together high-quality ML-oriented web apps. More recently, it has seen wider adoption in production environments by ever-larger organizations.

All of this means that there is no better time to pick up some experience with Streamlit. Fortunately, the basics of Streamlit are incredibly easy to learn, and for most tools, this will be more than you need!

In this series, we will introduce Streamlit by building a general knowledge Q&A interface. We will learn about key Streamlit components like write, text_input, container. How to use external libraries like Bootstrap to quickly create new app components. And use caching to speed up our app.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec?sk=ac5e0b7c39938f52162862411a66a58b

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:35 Streamlit Caching
06:56 Experimental Caching Primitives


Streamlit for ML #4 - Adding Bootstrap Components

▶ Streamlit for ML Part 5.1:
https://www.youtube.com/watch?v=SGazDb8o-to&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=5

Streamlit has proven itself as an incredibly popular tool for quickly putting together high-quality ML-oriented web apps. More recently, it has seen wider adoption in production environments by ever-larger organizations.

All of this means that there is no better time to pick up some experience with Streamlit. Fortunately, the basics of Streamlit are incredibly easy to learn, and for most tools, this will be more than you need!

In this series, we will introduce Streamlit by building a general knowledge Q&A interface. We will learn about key Streamlit components like write, text_input, container. How to use external libraries like Bootstrap to quickly create new app components. And use caching to speed up our app.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec?sk=ac5e0b7c39938f52162862411a66a58b

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:35 Streamlit Caching
06:56 Experimental Caching Primitives


Adding New Doc Stores to Haystack

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:15 Contributing or Testing
03:31 ODQA
06:20 What is Haystack?
08:13 Haystack QA Workflow
14:52 Contributing to Open Source
22:54 Haystack Doc Stores
26:09 Doc Store Core Methods
29:31 Final Notes, Contribute/Test


Streamlit for ML #5.1 - Custom React Components in Streamlit Setup

▶ Streamlit for ML Part 5.2:
https://www.youtube.com/watch?v=mxm8ihWoVbk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=6

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:19 Environment Setup
03:42 Starting with a Template
07:41 Naming for Card Component
11:31 Installing Node Packages
15:12 Running the Component


Streamlit for ML #5.2 - MUI Card Component Build

▶ Streamlit for ML Part 5.3:
https://www.youtube.com/watch?v=lZ2EaPUnV7k&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=7

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:59 Clearing Card Component
04:59 Building the Component
14:22 Pulling in MUI Code
24:08 Adding Roboto Font
26:05 Final Points


Streamlit for ML #5.3 - Publishing Components to Pip

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

❗ Python Packaging Video:
https://youtu.be/JkeNVaiUq_c

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:09 PyPI
02:41 Preparing for Distribution
05:43 Build React Component
06:39 Create Python Package
11:57 Pip Install
13:58 Ending


Train Sentence Transformers by Generating Queries (GenQ)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Fine-tuning effective dense retrieval models is challenging. Bi-encoders (sentence transformers) are the current best models for dense retrieval in semantic search. Unfortunately, they're also notoriously data-hungry models that typically require a particular type of labeled training data.

Hard problems like this attract attention. As expected, there is plenty of attention on building ever better techniques for training retrievers.

One of the most impressive is GenQ. This approach to building bi-encoder retrievers uses the latest text generation techniques to synthetically generate training data. In short, all we need are passages of text. The generation model then augments these passages with synthetic queries, giving us the exact format we need to train an effective bi-encoder model.

🌲 Pinecone article:
https://www.pinecone.io/learn/genq/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:32 Why GenQ?
02:23 GenQ Overview
04:28 Training Data
06:48 Asymmetric Semantic Search
07:54 T5 Query Generation
13:52 Finetuning Bi-encoders
16:02 GenQ Code Walkthrough
21:40 Finetuning Bi-encoder Walkthrough
26:48 Final Points


Testing the New Haystack Doc Store

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:19 Demo Start and Install
03:25 Initialization
06:30 Download and Write Documents
10:55 Extractive QA Pipeline
11:23 Fetch by ID
19:01 Metadata Filtering
22:24 Get All Documents


Is GPL the Future of Sentence Transformers? | Generative Pseudo-Labeling Deep Dive

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Training sentence transformers is hard; they need vast amounts of labeled data. On one hand, the internet is full of data, and, on the other, this data is *not* in the format we need. We usually need to use a supervised training method to train a high-performance bi-encoder (sentence transformer) model.

There is research producing techniques placing us ever closer to fine-tuning high-perfomance bi-encoder models with unlabeled text data. One of the most promising is GPL. At its core, GPL allows us to take unstructured text data and use it to build models that can understand this text. These models can then intelligently respond to natural language queries regarding this same text data.

It is a fascinating approach, with massive potential across innumerous use cases spanning all industries and borders. With that in mind, let's dive into the details of GPL and how we can implement it to build high-performance LMs with nothing more than plain text.

🌲 Pinecone article:
https://www.pinecone.io/learn/gpl/

🔗 Notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/nlp_course/gpl

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:08 Semantic Web and Other Uses
04:36 Why GPL?
07:31 How GPL Works
10:37 Query Generation
12:08 CORD-19 Dataset and Download
13:27 Query Generation Code
21:53 Query Generation is Not Perfect
22:39 Negative Mining
26:28 Negative Mining Implementation
27:21 Negative Mining Code
35:19 Pseudo-Labeling
35:55 Pseudo-Labeling Code
37:01 Importance of Pseudo-Labeling
41:20 Margin MSE Loss
43:40 MarginMSE Fine-tune Code
46:30 Choosing Number of Steps
48:54 Fast Evaluation
51:43 What's Next for Sentence Transformers?


Implementing Filters in the New Haystack Doc Store

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

Join me as I work through the final few PR issues on the latest Haystack document store, and figure out how Haystack's filter_utils work.

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:41 Filtering
05:36 Testing Existing Filter Utils
07:57 Making Sense of Filter Utils
10:35 Writing the First Filter
16:26 First Working Filter
18:24 Testing New Filters
21:27 Implementing in the Doc Store
24:02 Testing Pipeline Filters
27:11 Final Issue and Outro


Spotify's Podcast Search Explained

The market for podcasts has grown tremendously in recent years.

Driving the charge in podcast adoption is Spotify. In a few short years, they have become the undisputed leaders in podcasting. Despite only entering the game in 2018, by late 2021, Spotify had already usurped Apple, the long-reigning leader in podcasts, with more than 28M monthly podcast listeners.

To back their podcast investments, Spotify has worked on making the podcast experience as seamless and accessible as possible. From their all-in-one podcast creation app (Anchor) to podcast APIs and their latest natural language enabled podcast search.

Spotify’s natural language search for podcasts is a fascinating use case. In the past, users had to rely on keyword/term matching to find the podcast episodes they wanted. Now, they can search in natural language, in much the same way we might ask a real person where to find something.

In this video, we will take a look under the hood of Spotify's podcast search, and learn how to implement a similar system ourselves.

🌲 Pinecone article:
https://www.pinecone.io/learn/spotify-podcast-search

🔗 Code and tests:
https://github.com/pinecone-io/examples/tree/spotify-podcast-search/learn/search-in-wild/spotify-podcast-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:16 NLP in Semantic Search
08:35 Why Now?
09:29 Transformer Models
11:52 Sentence Transformers
13:12 Vector Search
15:56 How Spotify Built Podcast Search
17:35 Data Source, Fine-tuning, and Eval
22:58 Code Implementation, Dataset
24:44 Data Preparation
26:39 Query Generation
29:54 Fine-tuning a Podcast Model
41:40 Evaluation
48:05 Does it Scale?
49:00 Sharing Your Work


How to learn NLP for free

Knowing what to learn is one of the hardest parts about self-learning. Imagine being thrown into the wilderness and being told to find a specific landmark. Without a map you will end up wandering to wilderness with no better option than taking one step after another.

I spent a long time wandering step-by-step and eventually found my way into working with deep learning and NLP full-time.

Here I will share many of the resources I used or wish I had used in the past. You can this "curriculum" as a rough guideline in self-learning ML and working towards a full-time position.

ALL LINKS in article/friend link below:

📕 Medium article:
https://jamescalam.medium.com/the-self-taught-nlp-engineer-curriculum-c425c3fc3ff6

📖 Friend link:
https://jamescalam.medium.com/the-self-taught-nlp-engineer-curriculum-c425c3fc3ff6?sk=986263c644d9b36699d800713faa478a

---

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:53 ML 101 + Prerequisites
04:58 Sentdex + Neural Nets from Scratch
07:32 ML Coursera
09:31 100 Page ML Book
11:14 Applied ML + Daniel Bourke
13:17 Origin of Modern NLP
13:41 CS224N
14:44 NLP Specialization Coursera
15:57 Modern NLP + Transformers Intro
16:54 Transformer Courses
18:14 Doing Projects
19:18 Semantic + Vector Search
19:54 NLP for Semantic Search
20:44 Mining of Massive Datasets
22:27 Final Points


BERTopic Explained

90% of the world's data is unstructured. It is built by humans, for humans. That's great for human consumption, but it is *very* hard to organize when we begin dealing with the massive amounts of data abundant in today's information age.

Organization is complicated because unstructured text data is not intended to be understood by machines, and having humans process this abundance of data is wildly expensive and *very slow*.

Fortunately, there is light at the end of the tunnel. More and more of this unstructured text is becoming accessible and understood by machines. We can now search text based on *meaning*, identify the sentiment of text, extract entities, and much more.

Transformers are behind much of this. These transformers are (unfortunately) not Michael Bay's Autobots and Decepticons and (fortunately) not buzzing electrical boxes. Our NLP transformers lie somewhere in the middle, they're not sentient Autobots (yet), but they can understand language in a way that existed only in sci-fi until a short few years ago.

Machines with a human-like comprehension of language are pretty helpful for organizing masses of unstructured text data. In machine learning, we refer to this task as *topic modeling*, the automatic clustering of data into particular topics.

BERTopic takes advantage of the superior language capabilities of these (not yet sentient) transformer models and uses some other ML magic like UMAP and HDBSCAN (more on these later) to produce what is one of the most advanced techniques in language topic modeling today.

🌲 Pinecone article:
https://www.pinecone.io/learn/bertopic

🔗 Code notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/algos-and-libraries/bertopic

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:40 In this video
02:58 BERTopic Getting Started
08:48 BERTopic Components
15:21 Transformer Embedding
18:33 Dimensionality Reduction
25:07 UMAP
31:48 Clustering
37:22 c-TF-IDF
40:49 Custom BERTopic
44:04 Final Thoughts


Long Form Question Answering (LFQA) in Haystack

Question-Answering (QA) has exploded as a subdomain of Natural Language Processing (NLP) in the last few years. QA is a widely applicable use case in NLP yet was out of reach until the introduction of [transformer models](/learn/transformers/) in 2017.

Without transformer models, the level of language comprehension required to make something as complex as QA work simply was not possible.

Although QA is a complex topic, it comes from a simple idea. The automatic retrieval of information via a more human-like interaction. The task of information retrieval (IR) is performed by almost every organization in the world. Without other options, organizations rely on person-to-person IR and rigid keyword search tools. This haphazard approach to IR generates a lot of friction, particularly for larger organizations.

QA offers a solution to this problem. Rather than these documents being lost in an abyss, they can be stored within a space where an intelligent QA agent can access them. Unlike humans, our QA agent can scan millions of documents in seconds and return answers from these documents almost instantly.

With QA tools, employees can stop wasting time searching for snippets of information and focus on their *real*, value-adding tasks.

A small investment in QA is, for most organizations, a no-brainer.

🌲 Pinecone article:
https://www.pinecone.io/learn/haystack-lfqa

🔗 Code notebooks:
https://github.com/pinecone-io/examples/blob/master/integrations/haystack/haystack_lfqa.ipynb

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:20 Approaches to Question Answering
05:43 Components of QA Pipeline
08:58 LFQA Generator
09:40 Haystack Setup
10:32 Initialize Document Store
13:02 Getting Data
17:53 Indexing Embeddings
21:51 Initialize Generator
24:10 Asking Questions
26:12 Common Problems
29:32 Generator Memory
31:30 Few More Questions
34:54 Outro


New GPU-Acceleration for PyTorch on M1 Macs! + using with BERT

GPU-acceleration on Mac is finally here!

Today's deep learning models owe a great deal of their exponential performance gains to ever increasing model sizes. Those larger models require more computations to train and run.

These models are simply too big to be run on CPU hardware, which performs large step-by-step computations. Instead, they need massively parallel computations. That leaves us with either GPU or TPU hardware.

Our home PCs aren't coming with TPUs anytime soon, so we're left with the GPU option. GPUs use a highly parallel structure, originally designed to process images for visual heavy processes. They became essential components in gaming for rendering real-time 3D images.

GPUs are essential for the scale of today's models. Using CPUs makes many of these models too slow to be useful, which can make deep learning on M1 machines rather disappointing.

Fortunately, this is changing with the support of GPU on M1 machines beginning with PyTorch v1.12. In this video we will explain the new integration and how to implement it yourself.

📕  Article:
https://towardsdatascience.com/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1

📖  Friend Link (free access):
https://towardsdatascience.com/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1?sk=a88acd35f600858093c177b97d690b03

🔗  Code notebooks:
https://github.com/jamescalam/pytorch-mps

🤖  70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉  Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾  Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:34 PyTorch MPS
04:57 Installing ARM Python
09:09 Using PyTorch with GPU
12:14 BERT on PyTorch GPU
13:51 Best way to train LLMs on Mac
16:01 Buffer Size Bug
17:24 When we would use Mac M1 GPU


How to Build an AI-Powered Video Search App

Technology and culture have advanced and become ever more entangled. Some of the most significant technological breakthroughs are integrated so tightly into our culture that we never even notice they’re there.

One of those is AI-powered search. It powers your Google results, Netflix recommendations, and ads you see everywhere. It is being rapidly weaved throughout all aspects of our lives. Further, this is a new technology; its full potential is unknown.

This technology weaves directly into the cultural phenomenon of YouTube. Imagine a search engine like Google that allows you to rapidly access the billions of hours of YouTube content. There is no comparison to that level of highly engaging video content in the world.

🌲 Pinecone article:
https://www.pinecone.io/learn/youtube-search

🔗 Code:
https://github.com/pinecone-io/examples/tree/master/learn/projects/yt-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:56 YouTube Search App
04:43 Getting Data
07:58 Enhancing the Data
12:45 Scraping Other Metadata
14:52 Loading Data from Hugging Face
15:42 Index and Query the Data
20:43 Streamlit App Code


Using Semantic Search to Find GIFs

Vector search powers some of the most popular services in the world. It serves your Google results, delivers the best podcasts on Spotify, and accounts for at least 35% of consumer purchases on Amazon.

In this article, we will use vector search applied to language, called semantic search, to build a GIF search engine. Unlike more traditional search where we rely on keyword matching, semantic search enables search based on the human meaning behind text and images. That means we can find highly relevant GIFs with natural language prompts.

The pipeline for a project like this is simple, yet powerful. It can easily be adapted to tasks as diverse as video search or answering Super Bowl questions, or as we’ll see, finding GIFs.

🌲 Pinecone article:
https://www.pinecone.io/learn/gif-search

🔗 Code:
https://github.com/pinecone-io/examples/tree/master/learn/projects/gif-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:17 GIF Search Demo
01:56 Pipeline Overview
05:33 Data Preparation
08:17 Vector Database and Retriever
12:37 Querying
15:42 Streamlit App Code


How to Learn Data Science | ML | Programming

In this video I share five of the approaches/thoughts I have regarding learning, in particular for learning data science, machine learning, or programming.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:33 Scale of Theory vs. Applied
02:55 Shape of Learning
05:52 Courses vs. Projects
08:37 Open Source
10:44 Writing
12:44 Following Interests
15:42 Final Notes


Evaluation Measures for Search and Recommender Systems

In this video you will learn about popular offline metrics (evaluation measures) like Recall@K, Mean Reciprocal Rank (MRR), Mean Average Precision@K (MAP@K), and Normalized Discounted Cumulative Gain (NDCG@K). We will also demonstrate how each of these metrics can be replicated in Python.

Evaluation of information retrieval (IR) systems is critical to making well-informed design decisions. From search to recommendations, evaluation measures are paramount to understanding what does and does not work in retrieval.

Many big tech companies contribute much of their success to well-built IR systems. One of Amazon’s earliest iterations of the technology was reportedly driving more than 35% of their sales. Google attributes 70% of YouTube views to their IR recommender systems.

IR systems power some of the greatest companies in the world, and behind every successful IR system is a set of evaluation measures.

🌲 Pinecone article:
https://www.pinecone.io/learn/offline-evaluation

🔗 Code notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/algos-and-libraries/offline-evaluation

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:51 Offline Metrics
02:38 Dataset and Retrieval 101
06:08 Recall@K
07:57 Recall@K in Python
09:03 Disadvantages of Recall@K
10:21 MRR
13:32 MRR in Python
14:18 MAP@K
18:17 MAP@K in Python
19:27 NDCG@K
29:26 Pros and Cons of NDCG@K
29:48 Final Thoughts


How to build next-level Q&A with OpenAI

Walkthrough of the OpenAI x Pinecone Q&A app I built for a webinar with OpenAI. This is the coolest Q&A app I've ever built thanks to Pinecone vector search and OpenAI's incredible embeddings and generation endpoints.

LINKS:
🕹 App:
https://pinecone-io-playground-beyond-search-openaisrcserver-h65vzl.streamlitapp.com
👨💻 Code and Data:
https://github.com/pinecone-io/examples/tree/master/integrations/openai/beyond_search_webinar
OpenAI x Pinecone Webinar:
▶ https://www.youtube.com/watch?v=HtI9easWtAA

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


How to use Color Histograms for Image Retrieval

Browsing, searching, and retrieving images has never been easy. Traditionally, many technologies relied on manually appending metadata to images and searching via this metadata. This approach works for datasets with high-quality annotation, but most datasets are too large for manual annotation.

That means any large image dataset must rely on Content-Based Image Retrieval (CBIR). Search with CBIR focuses on comparing the *content* of an image rather than its metadata. Content can be color, shapes, textures – or with some of the latest advances in ML — the "human meaning" behind an image.

Color histograms represent one of the first CBIR techniques, allowing us to search through images based on their color profiles rather than metadata.

🌲 Pinecone article:
https://pinecone.io/learn/color-histograms

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:23 What are Color Histograms?
08:39 How to Built Color Histograms
16:56 Using OpenCV calcHist
20:36 Image Retrieval
27:37 Pros and Cons
30:40 Final Points


🤗 Hugging Face just released *Diffusers* - for models like DALL-E 2 and Imagen!

Hugging Face of transformer fame have created a whole new Python library for diffusion models! Diffusion models are a key component of models like OpenAI's DALL-E-2, Google's Imagen, and Midjourney's image generation service. HuggingFace Diffusers brings these models to a new level of accessibility (and open source!).

📕  Article:
https://towardsdatascience.com/hugging-face-just-released-the-diffusers-library-846f32845e65

📖  Friend Link (free access):
https://towardsdatascience.com/hugging-face-just-released-the-diffusers-library-846f32845e65?sk=9ec4027460defa1fd25178af9a55da13

🧨 Diffusers:
https://github.com/huggingface/diffusers

👾 Discord:
https://discord.gg/c5QtDB9RAP

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

00:00 What are Diffusers?
01:55 Getting started
04:20 Prompt engineering
09:34 Testing other diffusers


How I work from anywhere

Overview of how I deal with travel and work. Remote desk setup for staying as ergonomic and productive as possible, enjoy!

🔗 Links to products (mostly affiliate):
Laptop stand: https://amzn.to/3bZqMHM
Second screen: https://amzn.to/3w6IT5B
Cable bag (international): https://amzn.to/3QBH7S7
                              ... or UK: https://amzn.to/3ps5lT2
Peak Design backpacks: https://www.peakdesign.com/products/everyday-backpack

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Bag of *Visual* Words for Image Classification and Retrieval

In computer vision, bag of visual words (BoVW) is one of the pre-deep learning models used for building image embeddings. Allowing us to retrieve images from a database that are similar to another "query" image, perform object detection, and image classification.

🌲 Pinecone article:
https://www.pinecone.io/learn/bag-of-visual-words/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Fast intro to multi-modal ML with OpenAI's CLIP

OpenAI's CLIP is "multi-modal" model capable of understanding the relationships and concepts between both text and images. As we'll see, CLIP is very capable, and when used via the Hugging Face library, could not be easier to work with.

📕  Article:
https://towardsdatascience.com/quick-fire-guide-to-multi-modal-ml-with-openais-clip-2dad7e398ac0

📖  Friend Link (free access):
https://towardsdatascience.com/quick-fire-guide-to-multi-modal-ml-with-openais-clip-2dad7e398ac0?sk=89bb2d8b8e583ed109d8a05e00366645

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:15 What is CLIP?
02:13 Getting started
05:38 Creating text embeddings
07:23 Creating image embeddings
10:26 Embedding a lot of images
15:08 Text-image similarity search
21:38 Alternative image and text search


AlexNet and ImageNet Explained

Today’s deep learning revolution traces back to the 30th of September, 2012. On this day, a Convolutional Neural Network (CNN) called AlexNet won the ImageNet 2012 challenge. AlexNet didn’t just win; it dominated.

AlexNet was unlike the other competitors. This new model demonstrated unparalleled performance on the largest image dataset of the time, ImageNet. This event made AlexNet the first widely acknowledged, successful application of deep learning. It caught people’s attention with a 9.8 percentage point advantage over the nearest competitor.

Until this point, deep learning was a nice idea that most deemed as impractical. AlexNet showed that deep learning was more than a pipedream, and the authors showed the world how to make it practical. Yet, the surge of deep learning that followed was not fueled solely by AlexNet. Indeed, without the huge ImageNet dataset, there would have been no AlexNet.

The future of AI was to be built on the foundations set by the ImageNet challenge and the novel solutions that enabled the synergy between ImageNet and AlexNet.

🌲 Pinecone article:
https://pinecone.io/learn/imagenet

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:06 Birth of Deep Learning
02:52 ImageNet
07:56 Lack of Readiness for Big Datasets
09:57 ImageNet Challenge (ILSVRC)
11:47 AlexNet
19:30 PYTORCH IMPLEMENTATION
19:55 Data Preprocessing
27:06 Class Prediction with AlexNet
31:50 Goldfish Results
34:27 Closing Notes


Ultra-efficient Classifier Fine-tuning with Vector Search

Learn how to use vector search to create highly targeted training for any classification model using a final linear classification layer. Easily fine-tune models in 10 minutes with less than 100 labeled examples.

🌲 Pinecone article:
https://pinecone.io/learn/classifier-train-vector-search/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:14 Classification
02:49 Better Classifier Training
06:33 Classification as Vector Search
08:47 How Fine-tuning Works
10:50 Identifying Important Samples
12:39 CODE IMPLEMENTATION
13:13 Indexing
18:59 Fine-tuning the Classifier
27:37 Classifier Predictions
30:43 Closing Notes


Hugging Face Datasets #1 - Hosting your datasets

Introduction to Hugging Face datasets, how it works, and how to host your own simple datasets (JSONL, TSV, CSV, etc) for free via Hugging Face Datasets Hub

Warp download:
https://app.warp.dev/referral/7G3N39

Git LFS Install:
Mac:
$ brew install git-lfs
Debian/Ubuntu:
$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
$ sudo apt-get install git-lfs
Windows:
Get install from https://github.com/git-lfs/git-lfs/releases

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:36 Creating our own Datasets
08:29 Creating JSONL for Hugging Face
15:15 Uploading Datasets for Git
19:10 LFS for Large Files
21:56 Closing Notes


CLIP Explained | Multi-modal ML

Language models (LMs) can not rely on language alone. That is the idea behind the "Experience Grounds Language" paper, that proposes a framework to measure LMs' current and future progress. A key idea is that, beyond a certain threshold LMs need other forms of data, such as visual input.

The next step beyond well-known language models; BERT, GPT-3, and T5 is ”World Scope 3”. In World Scope 3, we move from large text-only datasets to large multi-modal datasets. That is, datasets containing information from multiple forms of media, like *both* images and text.

The world, both digital and real, is multi-modal. We perceive the world as an orchestra of language, imagery, video, smell, touch, and more. This chaotic ensemble produces an inner state, our "model" of the outside world.

AI must move in the same direction. Even specialist models that focus on language or vision must, at some point, have input from the other modalities. How can a model fully understand the concept of the word "person" without *seeing* a person?

OpenAI's Contrastive Learning In Pretraining (CLIP) is a world scope three model. It can comprehend concepts in both text and image and even connect concepts between the two modalities. In this video we will learn about multi-modality, how CLIP works, and how to use CLIP for different use cases like encoding, classification, and object detection.

🌲 Pinecone article:
https://pinecone.io/learn/clip/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Hugging Face Datasets #2 - Dataset Builder Scripts

How to work with dataset builder scripts, intro to the download manager, and Apache Arrow datatypes used in Hugging Face Datasets.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:49 Creating Compressed Files
02:41 Creating Dataset Build Script
04:49 Download Manager
08:59 Finishing Split Generator
10:13 Generate Examples Method
14:47 Add Dataset to Hugging Face
17:49 Apache Arrow Features
22:52 What's Next?


Zero-Shot Image Classification with OpenAI's CLIP

State-of-the-art (SotA) computer vision (CV) models are characterized by a *restricted* understanding of the visual world specific to their training data [1].

These models can perform *very well* on specific tasks and datasets, but they do not generalize well. They cannot handle new classes or images beyond the domain they have been trained with.

Ideally, a CV model should learn the contents of images without excessive focus on the specific labels it is initially trained to understand.

Fortunately, OpenAI's CLIP has proved itself as an incredibly flexible CV classification model that often requires *zero* retraining. In this chapter, we will explore CLIP in zero-shot image classification.

🌲 Pinecone article:
https://pinecone.io/learn/clip-classification/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


This is the worst AI ever

gpt4chan #4chan #ai GPT-4chan was trained on over 3 years of posts from 4chan's "politically incorrect" (/pol/) board. (and no ...


An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (Paper Explained)

ai #research #transformers Transformers are Ruining Convolutions. This paper, under review at ICLR, shows that given enough ...


NeuralHash is BROKEN - How to evade Apple's detection & craft hash collisions (w/ Open Source Code)

apple #icloud #neuralhash Send your Apple fanboy friends to prison with this one simple trick ;) We break Apple's NeuralHash ...


[ML News] Microsoft combines Images & Text | Meta makes artificial skin | Russians replicate DALL-E

mlnews #turing #reskin The latest and greatest from the Machine Learning world OUTLINE: 0:00 - Intro 0:15 - Sponsor: Weights ...


[ML News] BLOOM: 176B Open-Source | Chinese Brain-Scale Computer | Meta AI: No Language Left Behind

mlnews #bloom #ai Today we look at all the recent giant language models in the AI world! OUTLINE: 0:00 - Intro 0:55 - BLOOM: ...


Attention Is All You Need

https://arxiv.org/abs/1706.03762 Abstract: The dominant sequence transduction models are based on complex recurrent or ...


[ML News] GitHub Copilot - Copyright, GPL, Patents & more | Brickit LEGO app | Distill goes on break

copilot #copyright #gpl GitHub and OpenAI release Copilot, an AI-powered code autocomplete system that can generate entire ...


Perceiver: General Perception with Iterative Attention (Google DeepMind Research Paper Explained)

perceiver #deepmind #transformer Inspired by the fact that biological creatures attend to multiple modalities at the same time, ...


BLEURT: Learning Robust Metrics for Text Generation (Paper Explained)

Proper evaluation of text generation models, such as machine translation systems, requires expensive and slow human ...


Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment (Paper Explained)

neurips #peerreview #nips The peer-review system at Machine Learning conferences has come under much criticism over the last ...


Machine Learning PhD Survival Guide 2021 | Advice on Topic Selection, Papers, Conferences & more!

machinelearning #phd #howto This video is advice for new PhD students in the field of Machine Learning in 2021 and after.


[Classic] Deep Residual Learning for Image Recognition (Paper Explained)

ai #research #resnet ResNets are one of the cornerstones of modern Computer Vision. Before their invention, people were not ...


Involution: Inverting the Inherence of Convolution for Visual Recognition (Research Paper Explained)

involution #computervision #attention Convolutional Neural Networks (CNNs) have dominated computer vision for almost a ...


Did Google's LaMDA chatbot just become sentient?

lamda #google #ai Google engineer Blake Lemoine was put on leave after releasing proprietary information: An interview with the ...


[ML News] Google's 540B PaLM Language Model & OpenAI's DALL-E 2 Text-to-Image Revolution

mlnews #palm #dalle2 Google releases PaLM and OpenAI releases DALL-E 2 (and more news). Sponsor: Weights & BIases Start ...


The Man behind Stable Diffusion

stablediffusion #ai #stabilityai An interview with Emad Mostaque, founder of Stability AI. OUTLINE: 0:00 - Intro 1:30 - What is ...


NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)

nerf #neuralrendering #deeplearning View Synthesis is a tricky problem, especially when only given a sparse set of images as an ...


Curiosity-driven Exploration by Self-supervised Prediction

https://arxiv.org/abs/1705.05363 Authors: Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell Abstract: In many ...


mixup: Beyond Empirical Risk Minimization (Paper Explained)

Neural Networks often draw hard boundaries in high-dimensional space, which makes them very brittle. Mixup is a technique that ...


[ML News] Text-to-Image models are taking over! (Imagen, DALL-E 2, Midjourney, CogView 2 & more)

mlnews #dalle #imagen All things text-to-image models like DALL-E and Imagen! OUTLINE: 0:00 - Intro 0:30 - Imagen: Google's ...


Learning to summarize from human feedback (Paper Explained)

summarization #gpt3 #openai Text Summarization is a hard task, both in training and evaluation. Training is usually done ...


MuZero: Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

MuZero harnesses the power of AlphaZero, but without relying on an accurate environment model. This opens up planning-based ...


Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents

deepmind #rl #society This is an in-depth paper review, followed by an interview with the papers' authors! Society is ruled by ...


I'm taking a break

I'll be back, don't worry :) Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick YouTube: ...


Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

https://arxiv.org/abs/1502.03167 Abstract: Training Deep Neural Networks is complicated by the fact that the distribution of each ...


PAIR AI Explorables | Is the problem in the data? Examples on Fairness, Diversity, and Bias.

In the recurring debate about bias in Machine Learning models, there is a growing argument saying that "the problem is not in the ...


MLP-Mixer: An all-MLP Architecture for Vision (Machine Learning Research Paper Explained)

mixer #google #imagenet Convolutional Neural Networks have dominated computer vision for nearly 10 years, and that might ...


Grokking: Generalization beyond Overfitting on small algorithmic datasets (Paper Explained)

grokking #openai #deeplearning Grokking is a phenomenon when a neural network suddenly learns a pattern in the dataset and ...


DETR: End-to-End Object Detection with Transformers (Paper Explained)

Object detection in images is a notoriously hard task! Objects can be of a wide variety of classes, can be numerous or absent, they ...


Longformer: The Long-Document Transformer

The Longformer extends the Transformer by introducing sliding window attention and sparse global attention. This allows for the ...


OpenAI CLIP: ConnectingText and Images (Paper Explained)

ai #openai #technology Paper Title: Learning Transferable Visual Models From Natural Language Supervision CLIP trains on 400 ...


DINO: Emerging Properties in Self-Supervised Vision Transformers (Facebook AI Research Explained)

dino #facebook #selfsupervised Self-Supervised Learning is the final frontier in Representation Learning: Getting useful features ...


Reward Is Enough (Machine Learning Research Paper Explained)

reinforcementlearning #deepmind #agi What's the most promising path to creating Artificial General Intelligence (AGI)? This paper ...


[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)

ai #dqn #deepmind After the initial success of deep neural networks, especially convolutional neural networks on supervised ...


Does GPT-3 lie? - Misinformation and fear-mongering around the TruthfulQA dataset

gpt-3 #truth #conspiracy A new benchmark paper has created quite an uproar in the community. TruthfulQA is a dataset of 817 ...


Pretrained Transformers as Universal Computation Engines (Machine Learning Research Paper Explained)

universalcomputation #pretrainedtransformers #finetuning Large-scale pre-training and subsequent fine-tuning is a common ...


Neural Ordinary Differential Equations

https://arxiv.org/abs/1806.07366 Abstract: We introduce a new family of deep neural network models. Instead of specifying a ...


FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

FixMatch is a simple, yet surprisingly effective approach to semi-supervised learning. It combines two previous methods in a ...


Peer Review is still BROKEN! The NeurIPS 2021 Review Experiment (results are in)

neurips #peerreview #machinelearning A look at the results of the 2021 NeurIPS peer review experiment.


I'm out of Academia

machinelearning #ai #phd Done with my PhD in Machine Learning at ETH Zurich. On to new lands! Links: TabNine Code ...


[Rant] REVIEWER #2: How Peer Review is FAILING in Machine Learning

ai #research #peerreview Machine Learning research is in dire straits as more people flood into the field and competent reviewers ...


IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures

Policy Gradient RL on a massively distributed scale with theoretical guarantees! Abstract: In this work we aim to solve a large ...


[ML News] Stable Diffusion Takes Over! (Open Source AI Art)

stablediffusion #aiart #mlnews Stable Diffusion has been released and is riding a wave of creativity and collaboration. But not ...


How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit

ai #sparsity #gpu Sparsity is awesome, but only recently has it become possible to properly handle sparse models at good ...


NFNets: High-Performance Large-Scale Image Recognition Without Normalization (ML Paper Explained)

nfnets #deepmind #machinelearning Batch Normalization is a core component of modern deep learning. It enables training at ...


[ML News] DeepMind fails to get independence from Google

deepmind #google #mlnews DeepMind has reportedly failed to negotiate for greater independence from Google/Alphabet.


Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

ai #technology #switchtransformer Scale is the next frontier for AI. Google Brain uses sparsity and hard routing to massively ...


[ML News] AI models that write code (Copilot, CodeWhisperer, Pangu-Coder, etc.)

mlnews #ai #copilot OUTLINE: 0:00 - Intro 0:20 - Copilot Now Generally Available 3:20 - FOSS Org leaves GitHub 6:45 - Google's ...


[ML News] Meta's OPT 175B language model | DALL-E Mega is training | TorToiSe TTS fakes my voice

mlnews #dalle #gpt3 An inside look of what's happening in the ML world! Sponsor: Weights & Biases https://wandb.me/yannic ...


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

https://arxiv.org/abs/1810.04805 Abstract: We introduce a new language representation model called BERT, which stands for ...


Dream to Control: Learning Behaviors by Latent Imagination

Dreamer is a new RL agent by DeepMind that learns a continuous control task through forward-imagination in latent space.


Can Wikipedia Help Offline Reinforcement Learning? (Author Interview)

wikipedia #reinforcementlearning #languagemodels Original paper review here: https://youtu.be/XHGh19Hbx48 Machel Reid and ...


[Rant] The Male Only History of Deep Learning

This casting of our field in terms of ideological narrow-sighted group-think is disgusting. Keep Science about ideas!


What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (Paper Explained)

ai #research #machinelearning Online Reinforcement Learning is a flourishing field with countless methods for practitioners to ...


World Models

Authors: David Ha, Jürgen Schmidhuber Abstract: We explore building generative neural network models of popular ...


BERTology Meets Biology: Interpreting Attention in Protein Language Models (Paper Explained)

Proteins are the workhorses of almost all cellular functions and a core component of life. But despite their versatility, all proteins ...


[ML News] BLOOM: 176B Open-Source | Chinese Brain-Scale Computer | Meta AI: No Language Left Behind

mlnews #bloom #ai Today we look at all the recent giant language models in the AI world! OUTLINE: 0:00 - Intro 0:55 - BLOOM: ...


OpenAI Embeddings (and Controversy?!)

mlnews #openai #embeddings COMMENTS DIRECTLY FROM THE AUTHOR (thanks a lot for reaching out Arvind :) ): 1.


[ML News] Text-to-Image models are taking over! (Imagen, DALL-E 2, Midjourney, CogView 2 & more)

mlnews #dalle #imagen All things text-to-image models like DALL-E and Imagen! OUTLINE: 0:00 - Intro 0:30 - Imagen: Google's ...


Grokking: Generalization beyond Overfitting on small algorithmic datasets (Paper Explained)

grokking #openai #deeplearning Grokking is a phenomenon when a neural network suddenly learns a pattern in the dataset and ...


[ML News] Cedille French Language Model | YOU Search Engine | AI Finds Profitable MEME TOKENS

mlnews #cedille #wmt Only the greatest of news from the world of Machine Learning. OUTLINE: 0:00 - Sponsor: Weights & Biases ...


Learning model-based planning from scratch

https://arxiv.org/abs/1707.06170 Abstract: Conventional wisdom holds that model-based planning is a powerful approach to ...


DETR: End-to-End Object Detection with Transformers (Paper Explained)

Object detection in images is a notoriously hard task! Objects can be of a wide variety of classes, can be numerous or absent, they ...


GPT-NeoX-20B - Open-Source huge language model by EleutherAI (Interview w/ co-founder Connor Leahy)

eleuther #gptneo #gptj EleutherAI announces GPT-NeoX-20B, a 20 billion parameter open-source language model, inspired by ...


Do ImageNet Classifiers Generalize to ImageNet? (Paper Explained)

Has the world overfitted to ImageNet? What if we collect another dataset in exactly the same fashion? This paper gives a ...


The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Stunning evidence for the hypothesis that neural networks work so well because their random initialization almost certainly ...


[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)

ai #dqn #deepmind After the initial success of deep neural networks, especially convolutional neural networks on supervised ...


[ML News] This AI completes Wikipedia! Meta AI Sphere | Google Minerva | GPT-3 writes a paper

mlnews #ai #minerva This episode is all about models that reason. OUTLINE: 0:00 - Intro 0:35 - Meta AI learns Wikipedia citations ...


VirTex: Learning Visual Representations from Textual Annotations (Paper Explained)

Pre-training a CNN backbone for visual transfer learning has recently seen a big push into the direction of incorporating more data ...


CornerNet: Detecting Objects as Paired Keypoints (Paper Explained)

Many object detectors focus on locating the center of the object they want to find. However, this leaves them with the secondary ...


How far can we scale up? Deep Learning's Diminishing Returns (Article Review)

deeplearning #co2 #cost Deep Learning has achieved impressive results in the last years, not least due to the massive increases ...


Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Classic RL "stops" the world whenever the Agent computes a new action. This paper considers a more realistic scenario where ...


Adversarial Examples Are Not Bugs, They Are Features

Abstract: Adversarial examples have attracted significant attention in machine learning, but the reasons for their existence and ...


Research Conference ICML drops their acceptance rate | Area Chairs instructed to be more picky

icml #machinelearning #conference In a controversial move, ICML Area Chairs were instructed to raise the bar on acceptance to ...


NFNets: High-Performance Large-Scale Image Recognition Without Normalization (ML Paper Explained)

nfnets #deepmind #machinelearning Batch Normalization is a core component of modern deep learning. It enables training at ...


Dreamer v2: Mastering Atari with Discrete World Models (Machine Learning Research Paper Explained)

dreamer #deeprl #reinforcementlearning Model-Based Reinforcement Learning has been lagging behind Model-Free RL on Atari ...


Author Interview - VOS: Learning What You Don't Know by Virtual Outlier Synthesis

deeplearning #objectdetection #outliers An interview with the authors of "Virtual Outlier Synthesis". Watch the paper review video ...


A neurally plausible model learns successor representations in partially observable environments

Successor representations are a mid-point between model-based and model-free reinforcement learning. This paper learns ...


Turing-NLG, DeepSpeed and the ZeRO optimizer

Microsoft has trained a 17-billion parameter language model that achieves state-of-the-art perplexity. This video takes a look at ...


Reformer: The Efficient Transformer

The Transformer for the masses! Reformer solves the biggest problem with the famous Transformer model: Its huge resource ...


Neural Ordinary Differential Equations

https://arxiv.org/abs/1806.07366 Abstract: We introduce a new family of deep neural network models. Instead of specifying a ...


Jukebox: A Generative Model for Music (Paper Explained)

This generative model for music can make entire songs with remarkable quality and consistency. It can be conditioned on genre, ...


Backpropagation and the brain

Geoffrey Hinton and his co-authors describe a biologically plausible variant of backpropagation and report evidence that such an ...


My GitHub (Trash code I wrote during PhD)

phdlife #github #researchcode A brief browse through my public GitHub and musings about my old code.


Group Normalization (Paper Explained)

The dirty little secret of Batch Normalization is its intrinsic dependence on the training batch size. Group Normalization attempts to ...


Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (Paper Explained)

openai #vpt #minecraft Minecraft is one of the harder challenges any RL agent could face. Episodes are long, and the world is ...


Neural Architecture Search without Training (Paper Explained)

ai #research #machinelearning Neural Architecture Search is typically very slow and resource-intensive. A meta-controller has to ...


Does GPT-3 lie? - Misinformation and fear-mongering around the TruthfulQA dataset

gpt-3 #truth #conspiracy A new benchmark paper has created quite an uproar in the community. TruthfulQA is a dataset of 817 ...


Investigating Human Priors for Playing Video Games (Paper & Demo)

Why are humans so good at video games? Maybe it's because a lot of games are designed with humans in mind. What happens if ...


Learning Rate Grafting: Transferability of Optimizer Tuning (Machine Learning Research Paper Review)

grafting #adam #sgd The last years in deep learning research have given rise to a plethora of different optimization algorithms, ...


XLNet: Generalized Autoregressive Pretraining for Language Understanding

Abstract: With the capability of modeling bidirectional contexts, denoising autoencoding based pretraining like BERT achieves ...


Growing Neural Cellular Automata

The Game of Life on steroids! This model learns to grow complex patterns in an entirely local way. Each cell is trained to listen to ...


DINO: Emerging Properties in Self-Supervised Vision Transformers (Facebook AI Research Explained)

dino #facebook #selfsupervised Self-Supervised Learning is the final frontier in Representation Learning: Getting useful features ...


NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)

nerf #neuralrendering #deeplearning View Synthesis is a tricky problem, especially when only given a sparse set of images as an ...


State-of-Art-Reviewing: A Radical Proposal to Improve Scientific Publication

Peer Review is outdated and ineffective. SOAR is a new and revolutionary way to distribute scientific reviewing and scale to the ...


Efficient and Modular Implicit Differentiation (Machine Learning Research Paper Explained)

implicitfunction #jax #autodiff Many problems in Machine Learning involve loops of inner and outer optimization. Finding update ...


[ML News] DeepMind's Flamingo Image-Text model | Locked-Image Tuning | Jurassic X & MRKL

flamingo #mlnews #tech Your updates directly from the state of the art in Machine Learning! OUTLINE: 0:00 - Intro 0:30 ...


Imagination-Augmented Agents for Deep Reinforcement Learning

Commentary of https://arxiv.org/abs/1707.06203 Abstract We introduce Imagination-Augmented Agents (I2As), a novel ...


World Models

Authors: David Ha, Jürgen Schmidhuber Abstract: We explore building generative neural network models of popular ...


[Drama] Yann LeCun against Twitter on Dataset Bias

Yann LeCun points out an instance of dataset bias and proposes a sensible solution. People are not happy about it. Original ...


Radioactive data: tracing through training (Paper Explained)

ai #research #privacy Data is the modern gold. Neural classifiers can improve their performance by training on more data, but ...


Concept Learning with Energy-Based Models (Paper Explained)

This is a hard paper! Energy-functions are typically a mere afterthought in current machine learning. A core function of the Energy ...


[News] OpenAI Model Generates Python Code

This code completion engine can write an entire function from just the name! OpenAI demonstrates what happens when you learn ...


GPT-3: Language Models are Few-Shot Learners (Paper Explained)

gpt3 #openai #gpt-3 How far can you go with ONLY language modeling? Can a large enough language model perform NLP task ...


[ML News] OpenAI's Whisper | Meta Reads Brain Waves | AI Wins Art Fair, Annoys Humans

mlnews #openai #ai Everything important going on in the ML world right here! Sponsor: Paperspace ...


Accelerating Deep Learning by Focusing on the Biggest Losers

What if you could reduce the time your network trains by only training on the hard examples? This paper proposes to select ...


IT ARRIVED! YouTube sent me a package. (also: Limited Time Merch Deal)

LIMITED TIME MERCH DEAL: http://store.ykilcher.com Links: TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick ...