jamescalam/channel-metadata huggingface dataset:
Title: Building MLM Training Input Pipeline - Transformers From Scratch #3
Category: Education
Published: 2021-07-05 14:00:30 UTC
The input pipeline of our training process is the more complex part of the entire transformer build. It consists of us taking our raw OSCAR training data, transforming it, and preparing it for Masked-Language Modeling (MLM). Finally, we load our data into a DataLoader ready for training!

Part 1: https://youtu.be/GhGUZrcB-WM
Part 2: https://youtu.be/JIeAB8vvBQo
---
Part 4: https://youtu.be/35Pdoyi6ZoQ

📙 Medium article:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6

📖 Free link:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6?sk=9db6224efbd4ec6fd407a80b528e69b0

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Angular App Setup With Material - Stoic Q&A #5
Category: Science & Technology
Published: 2021-07-20 14:00:28 UTC
▶️ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

The fifth video in our Stoic Q&A series - setting up our Angular app with Angular Material.

Prerequisites:
Installation of Node.js and NPM - https://nodejs.org/en/
Angular - https://angular.io/guide/setup-local

👾 Discord
https://discord.gg/c5QtDB9RAP
Title: Training and Testing an Italian BERT - Transformers From Scratch #4
Category: Science & Technology
Published: 2021-07-06 13:00:03 UTC
We need two things for training, our DataLoader and a model. The DataLoader we have — but no model.

For training, we need a raw (not pre-trained) RobertaForMaskedLM. To create that, we first need to create a RoBERTa config object to describe the parameters we’d like to initialize FiliBERTo with.

Once we have our model, we set up our training loop and train!

Post-training, we'll test the model with Laura, who is Italian - and hope for the best.

Part 1: https://youtu.be/GhGUZrcB-WM
Part 2: https://youtu.be/JIeAB8vvBQo
Part 3: https://youtu.be/heTYbpr9mD8
---

📙 Medium article:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6?sk=9db6224efbd4ec6fd407a80b528e69b0

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:35 Review of Code
02:02 Config Object
06:28 Setup For Training
10:30 Training Loop
14:57 Dealing With CUDA Errors
16:17 Training Results
19:52 Loss
21:18 Fill-mask Pipeline For Testing
21:54 Testing With Laura
Title: Faiss - Introduction to Similarity Search
Category: Science & Technology
Published: 2021-07-13 15:00:19 UTC
Full Similarity Search Playlist:
https://www.youtube.com/watch?v=AY62z7HrghY&list=PLIUOU7oqGTLhlWpTz4NnuT3FekouIVlqc&index=1

Facebook AI Similarity Search (FAISS) is one of the most popular implementations of efficient similarity search, but what is it - and how can we use it?

What is it that makes FAISS special? How do we make the best use of this incredible tool?

Fortunately, it's a brilliantly simple process to get started with. And in this video, we'll explore some of the options FAISS provides, how they work, and - most importantly - how FAISS can make our semantic search faster.

🌲 Pinecone Article:
https://www.pinecone.io/learn/faiss-tutorial/

📊 Data:
https://github.com/jamescalam/data/tree/main/sentence_embeddings_15K

Notebook:
https://gist.github.com/jamescalam/7117aa92235a7f52141ad0654795aa48

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord
https://discord.gg/c5QtDB9RAP

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Why are there so many Tokenization methods in HF Transformers?
Category: Science & Technology
Published: 2021-07-27 14:00:10 UTC
HuggingFace's transformers library is the de-facto standard for NLP - used by practitioners worldwide, it's powerful, flexible, and easy to use. It achieves this through a fairly large (and complex) code-base, which has resulted in the question:

"Why are there so many tokenization methods in HuggingFace transformers?"

Tokenization is the process of encoding a string of text into transformer-readable token ID integers. In this video we cover five different methods for this - do these all produce the same output, or is there a difference between them?

📙 Medium article:
https://towardsdatascience.com/why-are-there-so-many-tokenization-methods-for-transformers-a340e493b3a8

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/why-are-there-so-many-tokenization-methods-for-transformers-a340e493b3a8?sk=4a7e8c88d331aef9103e153b5b799ff5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Choosing Indexes for Similarity Search (Faiss in Python)
Category: Science & Technology
Published: 2021-08-09 15:04:10 UTC
Facebook AI Similarity Search (Faiss) is a game-changer in the world of search. It allows us to efficiently search a huge range of media, from GIFs to articles - with incredible accuracy in sub-second timescales for billion+ size datasets.

The success in Faiss is due to many reasons. One of those, in particular, is its flexibility. Faiss recognizes that there is no 'one-size-fits-all' in similarity search.

Instead, Faiss comes with a wide range of search indexes - which we can mix and match to our choosing.

However, this great flexibility produces a question - how do we know which size fits our use case?

Which index do we choose? Should we use multiple indexes, or is one enough?

This video will explore the pros and cons of some of the most important indexes - Flat, LSH, HNSW, and IVF. We will learn how we decide which to use and the impact of parameters in each index to build some of the best indexes for semantic search.

🌲 Pinecone Article:
https://www.pinecone.io/learn/vector-indexes/

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

Download script for Sift1M dataset:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

Similarity Search Series:
https://www.youtube.com/playlist?list=PLIUOU7oqGTLhlWpTz4NnuT3FekouIVlqc

👾 Discord
https://discord.gg/c5QtDB9RAP

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Locality Sensitive Hashing (LSH) for Search with Shingling + MinHashing (Python)
Category: Science & Technology
Published: 2021-08-20 16:00:16 UTC
Locality sensitive hashing (LSH) is a widely popular technique used in approximate nearest neighbor (ANN) search. The solution to efficient similarity search is a profitable one - it is at the core of several billion (and even trillion) dollar companies.

LSH consists of a variety of different methods. In this video, we'll be covering the traditional approach - which consists of multiple steps - shingling, MinHashing, and the final banded LSH function.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing/

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
01:21 Overview
05:58 Shingling
08:45 Vocab
09:27 One-hot Encoding
11:10 MinHash
15:51 Signature Info
18:08 LSH
22:20 Tuning LSH
Title: How LSH Random Projection works in search (+Python)
Category: Science & Technology
Published: 2021-08-24 16:00:04 UTC
Locality sensitive hashing (LSH) is a widely popular technique used in approximate similarity search. The solution to efficient similarity search is a profitable one - it is at the core of several billion (and even trillion) dollar companies.

The problem with similarity search is scale. Many companies deal with millions-to-billions of data points every single day. Given a billion data points, is it feasible to compare all of them with every search?

Further, many companies are not performing single searches - Google deals with more than 3.8 million searches every minute.

Billions of data points combined with high-frequency searches are problematic - and we haven't considered the dimensionality nor the similarity function itself. Clearly, an exhaustive search across all data points is unrealistic for larger datasets.

The solution to searching impossibly huge datasets? Approximate search. Rather than exhaustively comparing every pair, we approximate - restricting the search scope only to high probability matches.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing-random-projection/

Download Sift1M:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

IndexLSH for Fast Similarity Search in Faiss:
https://youtu.be/ZLfdQq_u7Eo

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: IndexLSH for Fast Similarity Search in Faiss
Category: Science & Technology
Published: 2021-08-24 16:00:12 UTC
Faiss  -  or Facebook AI Similarity Search  -  is an open-source framework built for enabling similarity search.

Faiss has many super-efficient implementations of different indexes that we can use in similarity search. That long list of indexes includes IndexLSH - an easy-to-use implementation of everything we have covered so far in LSH.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing-random-projection/

Download Sift1M:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

How LSH Random Projection works in search (+Python):
https://youtu.be/8bOrMqEdfiQ

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Faiss - Vector Compression with PQ and IVFPQ (in Python)
Category: Science & Technology
Published: 2021-08-30 15:30:04 UTC
So far we’ve worked through the logic behind a simple, readable implementation of product quantization (PQ) in Python for semantic search. Realistically we wouldn’t use this because it is not optimized and we already have excellent implementations elsewhere. Instead, we would use a library like Faiss (Facebook AI Similarity Search) — or a production-ready service like Pinecone.

We’ll take a look at how we can build a PQ index in Faiss, and we’ll even take a look at combining PQ with an Inverted File (IVF) step to improve search speed.

Before we start, we need to get data. We will be using the Sift1M dataset. It can be downloaded and opened using this script:
https://gist.github.com/jamescalam/928a374b85daffa49a565f3dc18d059c#file-get_sift1m-ipynb

🌲 Pinecone article:
https://www.pinecone.io/learn/product-quantization/

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Product Quantization for Vector Similarity Search (+ Python)
Category: Science & Technology
Published: 2021-08-30 15:37:46 UTC
Vector similarity search can require huge amounts of memory. Indexes containing 1M dense vectors (a small dataset in today’s world) will often require several GBs of memory to store. When building recommendation systems or semantic search engines, this is not acceptable.

The problem of excessive memory usage is exasperated by high-dimensional data, and with ever-increasing dataset sizes, this can very quickly become unmanageable.

Product quantization (PQ) is a popular method for dramatically compressing high-dimensional vectors to use 97% less memory, and for making nearest-neighbor search speeds 5.5x faster in our tests.

A composite IVF+PQ index speeds up the search by another 16.5x without affecting accuracy, for a whopping total speed increase of 92x compared to non-quantized indexes.

🌲 Pinecone article:
https://www.pinecone.io/learn/product-quantization/

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free AI-Powered Code Refactoring with Sourcery: 
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff
Title: Composite Indexes and the Faiss Index Factory
Category: Science & Technology
Published: 2021-09-24 12:53:58 UTC
In the world of vector search, there are many indexing methods and vector processing techniques that allow us to prioritize between recall, latency, and memory usage.

Using specific methods such as IVF, PQ, or HNSW, we can often return good results. But for best performance we will usually want to use composite indexes.

We can view a composite index as a step-by-step process of vector transformations and one or more indexing methods. Allowing us to place multiple indexes and/or processing steps together to create our ‘ideal’ index.

For example, we can use an inverted file (IVF) index to reduce the scope of our search (increasing search speed), and then add a compression technique such as product quantization (PQ) to keep larger indexes within a reasonable size limit.

Where there is the ability to customize indexes, there is the risk of producing indexes with unnecessarily poor recall, latency, or memory usage.

We must know how composite indexes work if we want to build robust and high-performance vector similarity search applications. It is essential to understand where different indexes or vector transformations can be used — and when they are not needed.

Part 2: https://youtu.be/3Wqh4iUupbM

🌲 Pinecone article:
https://www.pinecone.io/learn/composite-indexes/

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

00:00 Intro
01:54 Composite Indexes
06:43 Faiss Index Factory
11:34 Why we use Index Factory
17:11 Outro
Title: Best Indexes for Similarity Search in Faiss
Category: Science & Technology
Published: 2021-09-24 12:54:07 UTC
In the world of vector search, there are many indexing methods and vector processing techniques that allow us to prioritize between recall, latency, and memory usage.

Using specific methods such as IVF, PQ, or HNSW, we can often return good results. But for best performance we will usually want to use composite indexes.

We can view a composite index as a step-by-step process of vector transformations and one or more indexing methods. Allowing us to place multiple indexes and/or processing steps together to create our ‘ideal’ index.

For example, we can use an inverted file (IVF) index to reduce the scope of our search (increasing search speed), and then add a compression technique such as product quantization (PQ) to keep larger indexes within a reasonable size limit.

Where there is the ability to customize indexes, there is the risk of producing indexes with unnecessarily poor recall, latency, or memory usage.

We must know how composite indexes work if we want to build robust and high-performance vector similarity search applications. It is essential to understand where different indexes or vector transformations can be used — and when they are not needed.

Part 1: https://youtu.be/GEhmmcx1lvM

🌲 Pinecone article:
https://www.pinecone.io/learn/composite-indexes/

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

00:00 Intro
00:30 IVFADC
03:30 IVFADC in Faiss
07:29 Multi-D-ADC
09:17 Multi-D-ADC in Faiss
14:43 IVF-HNSW
21:39 IVF-HNSW in Faiss
25:58 Outro
Title: How to Build a Bert WordPiece Tokenizer in Python and HuggingFace
Category: Science & Technology
Published: 2021-09-14 13:30:06 UTC
Building a transformer model from scratch can often be the only option for many more specific use cases. Although BERT and other transformer models have been pre-trained for a vast number of languages and domains, they do not cover everything.

Often, it is these less common use cases that stand to gain the most from having someone come along and build a specific transformer model. It could be for an uncommon language or less tech-savvy domain.

BERT is the most popular transformer for a wide range of language-based machine learning - from sentiment analysis to question and answering, BERT has enabled a diverse range of innovation across many borders and industries.

The first step for many in designing a new BERT model is the tokenizer. In this article, we'll take a look at the WordPiece tokenizer used by BERT - and see how we can build our own from scratch.

📕 Medium article:
https://towardsdatascience.com/how-to-build-a-wordpiece-tokenizer-for-bert-f505d97dddbb

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

🕹️ Free Article link (if you don't have Medium membership): 
https://towardsdatascience.com/how-to-build-a-wordpiece-tokenizer-for-bert-f505d97dddbb?sk=eea06e01c9faecd939e10589e9de1291
Title: Metadata Filtering for Vector Search + Latest Filter Tech
Category: Science & Technology
Published: 2021-09-20 14:04:27 UTC
Vector similarity search makes massive datasets searchable in fractions of a second. Yet despite the brilliance and utility of this technology, often what seem to be the most straightforward problems are the most difficult to solve. Such as filtering.

Filtering takes the top place in being seemingly simple — but actually incredibly complex. Applying fast-but-accurate filters when performing a vector search (ie, nearest-neighbor search) on massive datasets is a surprisingly stubborn problem.

This article explains the two common methods for adding filters to vector search, and their serious limitations. Then we will explore Pinecone’s solution to filtering in vector search.

📣 Get the API key!
https://www.pinecone.io/start/

🌲 Pinecone article:
https://www.pinecone.io/learn/vector-search-filtering/

00:00 Intro
00:24 Vector Search Recap
02:03 Why Filter?
02:56 Metadata Filtering 101
07:48 Pre-filtering
09:37 Post-filtering
11:30 Single-Stage Filtering
12:22 Vectors and Metadata Code
13:58 Connecting to Pinecone
14:55 Building Query Vector
16:47 Querying
21:37 First Filter
24:40 Adding More Conditions
27:03 Filtering with Numbers
30:55 Search Speed and Filtering
33:44 Outro
Title: Build NLP Pipelines with HuggingFace Datasets
Category: Science & Technology
Published: 2021-09-23 13:30:07 UTC
HF Datasets is an essential tool for NLP practitioners - hosting over 1.4K (mostly) high-quality language-focused datasets, and an easy-to-use treasure trove of functions for building efficient pre-processing pipelines.

In this article, we will take a look at the massive repository of datasets available, and explore some of the library's brilliant data processing capabilities.

📕 Medium article:
https://towardsdatascience.com/build-nlp-pipelines-with-huggingface-datasets-d597ff5f68ad

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Free Article Access (if you don't have Medium membership!): 
https://towardsdatascience.com/build-nlp-pipelines-with-huggingface-datasets-d597ff5f68ad?sk=948106e47e64bc3e9e8a1358b0568d48
Title: HNSW for Vector Search Explained and Implemented with Faiss (Python)
Category: Science & Technology
Published: 2021-10-05 13:00:23 UTC
Hierarchical Navigable Small World (HNSW) graphs are among the top-performing indexes for vector similarity search. HNSW is a hugely popular technology that time and time again produces state-of-the-art performance with super-fast search speeds and flawless recall — HNSW is not to be missed.

Despite being a popular and robust algorithm for approximate nearest neighbors (ANN) searches, understanding how it works is far from easy.

This video helps demystify HNSW and explains this intelligent algorithm in an easy-to-understand way. Towards the end of the video, we'll look at how to implement HNSW using Faiss and which parameter settings give us the performance we need.

🌲 Pinecone article:
https://www.pinecone.io/learn/hnsw/

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

00:00 Intro
00:41 Foundations of HNSW
08:41 How HNSW Works
16:38 The Basics of HNSW in Faiss
21:40 How Faiss Builds an HNSW Graph
26.49 Building the Best HNSW Index
33:33 Fine-tuning HNSW
34:30 Outro
Title: Intro to APIs in Python - API Series #1
Category: Science & Technology
Published: 2021-09-29 14:00:18 UTC
Taking those first steps into interacting with the web using Python can seem daunting - but it need not be. It is a surprisingly simple process, with well established rules and guidelines.

We'll cover the absolute essentials for getting started, including:

- Application Program Interfaces (APIs)
- Javascript Object Notation (JSON)
- Requests with Python
- Real world use-cases

📕 Article:
https://towardsdatascience.com/quick-fire-guide-to-apis-in-python-891dd98c8877

🎉 Sign-up For New Articles Every Week on Medium!
https://jamescalam.medium.com/subscribe (it's free!)
https://medium.com/@jamescalam/membership

📖 Free Access Link (if you don't have Medium membership): 
https://towardsdatascience.com/quick-fire-guide-to-apis-in-python-891dd98c8877?sk=7c159ba45154db23abcc6a7f9de4f910

Geocoding Docs:
https://developers.google.com/maps/documentation/geocoding/cloud-setup

GitHub Docs:
https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

00:00 Intro
00:20 What is an API?
01:47 RESTful APIs
05:26 API Methods
07:20 HTTP Codes (200s)
08:14 HTTP Codes (400s)
10:00 JSON Format
11:21 Talking to APIs in Python
14:30 Google Geocoding API
22:08 GitHub API
27:48 Outro
Title: Intro to Dense Vectors for NLP and Vision
Category: Science & Technology
Published: 2021-10-12 17:47:15 UTC
There is perhaps no greater component to the success of modern Natural Language Processing (NLP) technology than vector representations of language. The meteoric early 2010s rise of NLP was ignited with the introduction of word2vec by a team lead by Tomáš Mikolov in 2013.

Word2vec is one of the most iconic and earliest examples of dense vectors representing text. But since the days of word2vec, developments in representing language have advanced at ludicrous speeds.

This video will explore *why* we use dense vectors — and some of the best approaches to building dense vectors available today.

🌲 Pinecone article:
https://www.pinecone.io/learn/dense-vector-embeddings-nlp/

00:00 Intro
01:50 Why Dense Vectors?
03:55 Word2vec and Representing Meaning
08:40 Sentence Transformers
09:58 Sentence Transformers in Python
15:08 Question-Answering
18:18 DPR in Python
29:55 Vision Transformers
33:22 OpenAI's CLIP in Python
42:49 Review and What's Next