Streamlit for ML #4 - Adding Bootstrap Components

▶ Streamlit for ML Part 5.1:
https://www.youtube.com/watch?v=SGazDb8o-to&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=5

Streamlit has proven itself as an incredibly popular tool for quickly putting together high-quality ML-oriented web apps. More recently, it has seen wider adoption in production environments by ever-larger organizations.

All of this means that there is no better time to pick up some experience with Streamlit. Fortunately, the basics of Streamlit are incredibly easy to learn, and for most tools, this will be more than you need!

In this series, we will introduce Streamlit by building a general knowledge Q&A interface. We will learn about key Streamlit components like write, text_input, container. How to use external libraries like Bootstrap to quickly create new app components. And use caching to speed up our app.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
https://towardsdatascience.com/getting-started-with-streamlit-for-nlp-75fe463821ec?sk=ac5e0b7c39938f52162862411a66a58b

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:35 Streamlit Caching
06:56 Experimental Caching Primitives


Adding New Doc Stores to Haystack

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:15 Contributing or Testing
03:31 ODQA
06:20 What is Haystack?
08:13 Haystack QA Workflow
14:52 Contributing to Open Source
22:54 Haystack Doc Stores
26:09 Doc Store Core Methods
29:31 Final Notes, Contribute/Test


Streamlit for ML #5.1 - Custom React Components in Streamlit Setup

▶ Streamlit for ML Part 5.2:
https://www.youtube.com/watch?v=mxm8ihWoVbk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=6

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:19 Environment Setup
03:42 Starting with a Template
07:41 Naming for Card Component
11:31 Installing Node Packages
15:12 Running the Component


Streamlit for ML #5.2 - MUI Card Component Build

▶ Streamlit for ML Part 5.3:
https://www.youtube.com/watch?v=lZ2EaPUnV7k&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=7

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:59 Clearing Card Component
04:59 Building the Component
14:22 Pulling in MUI Code
24:08 Adding Roboto Font
26:05 Final Points


Streamlit for ML #5.3 - Publishing Components to Pip

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

There are plenty of prebuilt components designed by Streamlit themselves, and if you can't find what you need, there are even community-built components.

If you're still stuck, and there is just no component that covers what you need, we can build our own custom components.

To do this we do need to start playing with the lower-level web technologies that Streamlit itself is built upon. So it isn't as simple as using a prebuilt component. However, thanks to pre-made templates, it isn't too hard to create a new component.

In this sub-series, we'll learn exactly how to create custom components. We'll focus on designing an interactive card component using Material UI design elements.

❗ Python Packaging Video:
https://youtu.be/JkeNVaiUq_c

▶ Streamlit for ML Playlist:
https://www.youtube.com/watch?v=JLKUV-LiXjk&list=PLIUOU7oqGTLg5ssYxPGWaci6695wtosGw&index=1

📕 Article:
Coming soon

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

📖 Friend link to article:
Coming soon

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:09 PyPI
02:41 Preparing for Distribution
05:43 Build React Component
06:39 Create Python Package
11:57 Pip Install
13:58 Ending


Train Sentence Transformers by Generating Queries (GenQ)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Fine-tuning effective dense retrieval models is challenging. Bi-encoders (sentence transformers) are the current best models for dense retrieval in semantic search. Unfortunately, they're also notoriously data-hungry models that typically require a particular type of labeled training data.

Hard problems like this attract attention. As expected, there is plenty of attention on building ever better techniques for training retrievers.

One of the most impressive is GenQ. This approach to building bi-encoder retrievers uses the latest text generation techniques to synthetically generate training data. In short, all we need are passages of text. The generation model then augments these passages with synthetic queries, giving us the exact format we need to train an effective bi-encoder model.

🌲 Pinecone article:
https://www.pinecone.io/learn/genq/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:32 Why GenQ?
02:23 GenQ Overview
04:28 Training Data
06:48 Asymmetric Semantic Search
07:54 T5 Query Generation
13:52 Finetuning Bi-encoders
16:02 GenQ Code Walkthrough
21:40 Finetuning Bi-encoder Walkthrough
26:48 Final Points


Testing the New Haystack Doc Store

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:19 Demo Start and Install
03:25 Initialization
06:30 Download and Write Documents
10:55 Extractive QA Pipeline
11:23 Fetch by ID
19:01 Metadata Filtering
22:24 Get All Documents


Is GPL the Future of Sentence Transformers? | Generative Pseudo-Labeling Deep Dive

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Training sentence transformers is hard; they need vast amounts of labeled data. On one hand, the internet is full of data, and, on the other, this data is *not* in the format we need. We usually need to use a supervised training method to train a high-performance bi-encoder (sentence transformer) model.

There is research producing techniques placing us ever closer to fine-tuning high-perfomance bi-encoder models with unlabeled text data. One of the most promising is GPL. At its core, GPL allows us to take unstructured text data and use it to build models that can understand this text. These models can then intelligently respond to natural language queries regarding this same text data.

It is a fascinating approach, with massive potential across innumerous use cases spanning all industries and borders. With that in mind, let's dive into the details of GPL and how we can implement it to build high-performance LMs with nothing more than plain text.

🌲 Pinecone article:
https://www.pinecone.io/learn/gpl/

🔗 Notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/nlp_course/gpl

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:08 Semantic Web and Other Uses
04:36 Why GPL?
07:31 How GPL Works
10:37 Query Generation
12:08 CORD-19 Dataset and Download
13:27 Query Generation Code
21:53 Query Generation is Not Perfect
22:39 Negative Mining
26:28 Negative Mining Implementation
27:21 Negative Mining Code
35:19 Pseudo-Labeling
35:55 Pseudo-Labeling Code
37:01 Importance of Pseudo-Labeling
41:20 Margin MSE Loss
43:40 MarginMSE Fine-tune Code
46:30 Choosing Number of Steps
48:54 Fast Evaluation
51:43 What's Next for Sentence Transformers?


Implementing Filters in the New Haystack Doc Store

🥳  Released with Haystack v1.3! Install direct from PyPI with:

pip install 'farm-haystack[pinecone]'

Join me as I work through the final few PR issues on the latest Haystack document store, and figure out how Haystack's filter_utils work.

PR:
https://github.com/deepset-ai/haystack/pull/2254

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:41 Filtering
05:36 Testing Existing Filter Utils
07:57 Making Sense of Filter Utils
10:35 Writing the First Filter
16:26 First Working Filter
18:24 Testing New Filters
21:27 Implementing in the Doc Store
24:02 Testing Pipeline Filters
27:11 Final Issue and Outro


Spotify's Podcast Search Explained

The market for podcasts has grown tremendously in recent years.

Driving the charge in podcast adoption is Spotify. In a few short years, they have become the undisputed leaders in podcasting. Despite only entering the game in 2018, by late 2021, Spotify had already usurped Apple, the long-reigning leader in podcasts, with more than 28M monthly podcast listeners.

To back their podcast investments, Spotify has worked on making the podcast experience as seamless and accessible as possible. From their all-in-one podcast creation app (Anchor) to podcast APIs and their latest natural language enabled podcast search.

Spotify’s natural language search for podcasts is a fascinating use case. In the past, users had to rely on keyword/term matching to find the podcast episodes they wanted. Now, they can search in natural language, in much the same way we might ask a real person where to find something.

In this video, we will take a look under the hood of Spotify's podcast search, and learn how to implement a similar system ourselves.

🌲 Pinecone article:
https://www.pinecone.io/learn/spotify-podcast-search

🔗 Code and tests:
https://github.com/pinecone-io/examples/tree/spotify-podcast-search/learn/search-in-wild/spotify-podcast-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:16 NLP in Semantic Search
08:35 Why Now?
09:29 Transformer Models
11:52 Sentence Transformers
13:12 Vector Search
15:56 How Spotify Built Podcast Search
17:35 Data Source, Fine-tuning, and Eval
22:58 Code Implementation, Dataset
24:44 Data Preparation
26:39 Query Generation
29:54 Fine-tuning a Podcast Model
41:40 Evaluation
48:05 Does it Scale?
49:00 Sharing Your Work


How to learn NLP for free

Knowing what to learn is one of the hardest parts about self-learning. Imagine being thrown into the wilderness and being told to find a specific landmark. Without a map you will end up wandering to wilderness with no better option than taking one step after another.

I spent a long time wandering step-by-step and eventually found my way into working with deep learning and NLP full-time.

Here I will share many of the resources I used or wish I had used in the past. You can this "curriculum" as a rough guideline in self-learning ML and working towards a full-time position.

ALL LINKS in article/friend link below:

📕 Medium article:
https://jamescalam.medium.com/the-self-taught-nlp-engineer-curriculum-c425c3fc3ff6

📖 Friend link:
https://jamescalam.medium.com/the-self-taught-nlp-engineer-curriculum-c425c3fc3ff6?sk=986263c644d9b36699d800713faa478a

---

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:53 ML 101 + Prerequisites
04:58 Sentdex + Neural Nets from Scratch
07:32 ML Coursera
09:31 100 Page ML Book
11:14 Applied ML + Daniel Bourke
13:17 Origin of Modern NLP
13:41 CS224N
14:44 NLP Specialization Coursera
15:57 Modern NLP + Transformers Intro
16:54 Transformer Courses
18:14 Doing Projects
19:18 Semantic + Vector Search
19:54 NLP for Semantic Search
20:44 Mining of Massive Datasets
22:27 Final Points


BERTopic Explained

90% of the world's data is unstructured. It is built by humans, for humans. That's great for human consumption, but it is *very* hard to organize when we begin dealing with the massive amounts of data abundant in today's information age.

Organization is complicated because unstructured text data is not intended to be understood by machines, and having humans process this abundance of data is wildly expensive and *very slow*.

Fortunately, there is light at the end of the tunnel. More and more of this unstructured text is becoming accessible and understood by machines. We can now search text based on *meaning*, identify the sentiment of text, extract entities, and much more.

Transformers are behind much of this. These transformers are (unfortunately) not Michael Bay's Autobots and Decepticons and (fortunately) not buzzing electrical boxes. Our NLP transformers lie somewhere in the middle, they're not sentient Autobots (yet), but they can understand language in a way that existed only in sci-fi until a short few years ago.

Machines with a human-like comprehension of language are pretty helpful for organizing masses of unstructured text data. In machine learning, we refer to this task as *topic modeling*, the automatic clustering of data into particular topics.

BERTopic takes advantage of the superior language capabilities of these (not yet sentient) transformer models and uses some other ML magic like UMAP and HDBSCAN (more on these later) to produce what is one of the most advanced techniques in language topic modeling today.

🌲 Pinecone article:
https://www.pinecone.io/learn/bertopic

🔗 Code notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/algos-and-libraries/bertopic

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:40 In this video
02:58 BERTopic Getting Started
08:48 BERTopic Components
15:21 Transformer Embedding
18:33 Dimensionality Reduction
25:07 UMAP
31:48 Clustering
37:22 c-TF-IDF
40:49 Custom BERTopic
44:04 Final Thoughts


Long Form Question Answering (LFQA) in Haystack

Question-Answering (QA) has exploded as a subdomain of Natural Language Processing (NLP) in the last few years. QA is a widely applicable use case in NLP yet was out of reach until the introduction of [transformer models](/learn/transformers/) in 2017.

Without transformer models, the level of language comprehension required to make something as complex as QA work simply was not possible.

Although QA is a complex topic, it comes from a simple idea. The automatic retrieval of information via a more human-like interaction. The task of information retrieval (IR) is performed by almost every organization in the world. Without other options, organizations rely on person-to-person IR and rigid keyword search tools. This haphazard approach to IR generates a lot of friction, particularly for larger organizations.

QA offers a solution to this problem. Rather than these documents being lost in an abyss, they can be stored within a space where an intelligent QA agent can access them. Unlike humans, our QA agent can scan millions of documents in seconds and return answers from these documents almost instantly.

With QA tools, employees can stop wasting time searching for snippets of information and focus on their *real*, value-adding tasks.

A small investment in QA is, for most organizations, a no-brainer.

🌲 Pinecone article:
https://www.pinecone.io/learn/haystack-lfqa

🔗 Code notebooks:
https://github.com/pinecone-io/examples/blob/master/integrations/haystack/haystack_lfqa.ipynb

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:20 Approaches to Question Answering
05:43 Components of QA Pipeline
08:58 LFQA Generator
09:40 Haystack Setup
10:32 Initialize Document Store
13:02 Getting Data
17:53 Indexing Embeddings
21:51 Initialize Generator
24:10 Asking Questions
26:12 Common Problems
29:32 Generator Memory
31:30 Few More Questions
34:54 Outro


New GPU-Acceleration for PyTorch on M1 Macs! + using with BERT

GPU-acceleration on Mac is finally here!

Today's deep learning models owe a great deal of their exponential performance gains to ever increasing model sizes. Those larger models require more computations to train and run.

These models are simply too big to be run on CPU hardware, which performs large step-by-step computations. Instead, they need massively parallel computations. That leaves us with either GPU or TPU hardware.

Our home PCs aren't coming with TPUs anytime soon, so we're left with the GPU option. GPUs use a highly parallel structure, originally designed to process images for visual heavy processes. They became essential components in gaming for rendering real-time 3D images.

GPUs are essential for the scale of today's models. Using CPUs makes many of these models too slow to be useful, which can make deep learning on M1 machines rather disappointing.

Fortunately, this is changing with the support of GPU on M1 machines beginning with PyTorch v1.12. In this video we will explain the new integration and how to implement it yourself.

📕  Article:
https://towardsdatascience.com/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1

📖  Friend Link (free access):
https://towardsdatascience.com/gpu-acceleration-comes-to-pytorch-on-m1-macs-195c399efcc1?sk=a88acd35f600858093c177b97d690b03

🔗  Code notebooks:
https://github.com/jamescalam/pytorch-mps

🤖  70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉  Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾  Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:34 PyTorch MPS
04:57 Installing ARM Python
09:09 Using PyTorch with GPU
12:14 BERT on PyTorch GPU
13:51 Best way to train LLMs on Mac
16:01 Buffer Size Bug
17:24 When we would use Mac M1 GPU


How to Build an AI-Powered Video Search App

Technology and culture have advanced and become ever more entangled. Some of the most significant technological breakthroughs are integrated so tightly into our culture that we never even notice they’re there.

One of those is AI-powered search. It powers your Google results, Netflix recommendations, and ads you see everywhere. It is being rapidly weaved throughout all aspects of our lives. Further, this is a new technology; its full potential is unknown.

This technology weaves directly into the cultural phenomenon of YouTube. Imagine a search engine like Google that allows you to rapidly access the billions of hours of YouTube content. There is no comparison to that level of highly engaging video content in the world.

🌲 Pinecone article:
https://www.pinecone.io/learn/youtube-search

🔗 Code:
https://github.com/pinecone-io/examples/tree/master/learn/projects/yt-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
02:56 YouTube Search App
04:43 Getting Data
07:58 Enhancing the Data
12:45 Scraping Other Metadata
14:52 Loading Data from Hugging Face
15:42 Index and Query the Data
20:43 Streamlit App Code


Using Semantic Search to Find GIFs

Vector search powers some of the most popular services in the world. It serves your Google results, delivers the best podcasts on Spotify, and accounts for at least 35% of consumer purchases on Amazon.

In this article, we will use vector search applied to language, called semantic search, to build a GIF search engine. Unlike more traditional search where we rely on keyword matching, semantic search enables search based on the human meaning behind text and images. That means we can find highly relevant GIFs with natural language prompts.

The pipeline for a project like this is simple, yet powerful. It can easily be adapted to tasks as diverse as video search or answering Super Bowl questions, or as we’ll see, finding GIFs.

🌲 Pinecone article:
https://www.pinecone.io/learn/gif-search

🔗 Code:
https://github.com/pinecone-io/examples/tree/master/learn/projects/gif-search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:17 GIF Search Demo
01:56 Pipeline Overview
05:33 Data Preparation
08:17 Vector Database and Retriever
12:37 Querying
15:42 Streamlit App Code


How to Learn Data Science | ML | Programming

In this video I share five of the approaches/thoughts I have regarding learning, in particular for learning data science, machine learning, or programming.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:33 Scale of Theory vs. Applied
02:55 Shape of Learning
05:52 Courses vs. Projects
08:37 Open Source
10:44 Writing
12:44 Following Interests
15:42 Final Notes


Evaluation Measures for Search and Recommender Systems

In this video you will learn about popular offline metrics (evaluation measures) like Recall@K, Mean Reciprocal Rank (MRR), Mean Average Precision@K (MAP@K), and Normalized Discounted Cumulative Gain (NDCG@K). We will also demonstrate how each of these metrics can be replicated in Python.

Evaluation of information retrieval (IR) systems is critical to making well-informed design decisions. From search to recommendations, evaluation measures are paramount to understanding what does and does not work in retrieval.

Many big tech companies contribute much of their success to well-built IR systems. One of Amazon’s earliest iterations of the technology was reportedly driving more than 35% of their sales. Google attributes 70% of YouTube views to their IR recommender systems.

IR systems power some of the greatest companies in the world, and behind every successful IR system is a set of evaluation measures.

🌲 Pinecone article:
https://www.pinecone.io/learn/offline-evaluation

🔗 Code notebooks:
https://github.com/pinecone-io/examples/tree/master/learn/algos-and-libraries/offline-evaluation

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:51 Offline Metrics
02:38 Dataset and Retrieval 101
06:08 Recall@K
07:57 Recall@K in Python
09:03 Disadvantages of Recall@K
10:21 MRR
13:32 MRR in Python
14:18 MAP@K
18:17 MAP@K in Python
19:27 NDCG@K
29:26 Pros and Cons of NDCG@K
29:48 Final Thoughts


How to build next-level Q&A with OpenAI

Walkthrough of the OpenAI x Pinecone Q&A app I built for a webinar with OpenAI. This is the coolest Q&A app I've ever built thanks to Pinecone vector search and OpenAI's incredible embeddings and generation endpoints.

LINKS:
🕹 App:
https://pinecone-io-playground-beyond-search-openaisrcserver-h65vzl.streamlitapp.com
👨💻 Code and Data:
https://github.com/pinecone-io/examples/tree/master/integrations/openai/beyond_search_webinar
OpenAI x Pinecone Webinar:
▶ https://www.youtube.com/watch?v=HtI9easWtAA

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


How to use Color Histograms for Image Retrieval

Browsing, searching, and retrieving images has never been easy. Traditionally, many technologies relied on manually appending metadata to images and searching via this metadata. This approach works for datasets with high-quality annotation, but most datasets are too large for manual annotation.

That means any large image dataset must rely on Content-Based Image Retrieval (CBIR). Search with CBIR focuses on comparing the *content* of an image rather than its metadata. Content can be color, shapes, textures – or with some of the latest advances in ML — the "human meaning" behind an image.

Color histograms represent one of the first CBIR techniques, allowing us to search through images based on their color profiles rather than metadata.

🌲 Pinecone article:
https://pinecone.io/learn/color-histograms

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:23 What are Color Histograms?
08:39 How to Built Color Histograms
16:56 Using OpenCV calcHist
20:36 Image Retrieval
27:37 Pros and Cons
30:40 Final Points


🤗 Hugging Face just released *Diffusers* - for models like DALL-E 2 and Imagen!

Hugging Face of transformer fame have created a whole new Python library for diffusion models! Diffusion models are a key component of models like OpenAI's DALL-E-2, Google's Imagen, and Midjourney's image generation service. HuggingFace Diffusers brings these models to a new level of accessibility (and open source!).

📕  Article:
https://towardsdatascience.com/hugging-face-just-released-the-diffusers-library-846f32845e65

📖  Friend Link (free access):
https://towardsdatascience.com/hugging-face-just-released-the-diffusers-library-846f32845e65?sk=9ec4027460defa1fd25178af9a55da13

🧨 Diffusers:
https://github.com/huggingface/diffusers

👾 Discord:
https://discord.gg/c5QtDB9RAP

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

00:00 What are Diffusers?
01:55 Getting started
04:20 Prompt engineering
09:34 Testing other diffusers


How I work from anywhere

Overview of how I deal with travel and work. Remote desk setup for staying as ergonomic and productive as possible, enjoy!

🔗 Links to products (mostly affiliate):
Laptop stand: https://amzn.to/3bZqMHM
Second screen: https://amzn.to/3w6IT5B
Cable bag (international): https://amzn.to/3QBH7S7
                              ... or UK: https://amzn.to/3ps5lT2
Peak Design backpacks: https://www.peakdesign.com/products/everyday-backpack

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Bag of *Visual* Words for Image Classification and Retrieval

In computer vision, bag of visual words (BoVW) is one of the pre-deep learning models used for building image embeddings. Allowing us to retrieve images from a database that are similar to another "query" image, perform object detection, and image classification.

🌲 Pinecone article:
https://www.pinecone.io/learn/bag-of-visual-words/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Fast intro to multi-modal ML with OpenAI's CLIP

OpenAI's CLIP is "multi-modal" model capable of understanding the relationships and concepts between both text and images. As we'll see, CLIP is very capable, and when used via the Hugging Face library, could not be easier to work with.

📕  Article:
https://towardsdatascience.com/quick-fire-guide-to-multi-modal-ml-with-openais-clip-2dad7e398ac0

📖  Friend Link (free access):
https://towardsdatascience.com/quick-fire-guide-to-multi-modal-ml-with-openais-clip-2dad7e398ac0?sk=89bb2d8b8e583ed109d8a05e00366645

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:15 What is CLIP?
02:13 Getting started
05:38 Creating text embeddings
07:23 Creating image embeddings
10:26 Embedding a lot of images
15:08 Text-image similarity search
21:38 Alternative image and text search


AlexNet and ImageNet Explained

Today’s deep learning revolution traces back to the 30th of September, 2012. On this day, a Convolutional Neural Network (CNN) called AlexNet won the ImageNet 2012 challenge. AlexNet didn’t just win; it dominated.

AlexNet was unlike the other competitors. This new model demonstrated unparalleled performance on the largest image dataset of the time, ImageNet. This event made AlexNet the first widely acknowledged, successful application of deep learning. It caught people’s attention with a 9.8 percentage point advantage over the nearest competitor.

Until this point, deep learning was a nice idea that most deemed as impractical. AlexNet showed that deep learning was more than a pipedream, and the authors showed the world how to make it practical. Yet, the surge of deep learning that followed was not fueled solely by AlexNet. Indeed, without the huge ImageNet dataset, there would have been no AlexNet.

The future of AI was to be built on the foundations set by the ImageNet challenge and the novel solutions that enabled the synergy between ImageNet and AlexNet.

🌲 Pinecone article:
https://pinecone.io/learn/imagenet

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:06 Birth of Deep Learning
02:52 ImageNet
07:56 Lack of Readiness for Big Datasets
09:57 ImageNet Challenge (ILSVRC)
11:47 AlexNet
19:30 PYTORCH IMPLEMENTATION
19:55 Data Preprocessing
27:06 Class Prediction with AlexNet
31:50 Goldfish Results
34:27 Closing Notes


Ultra-efficient Classifier Fine-tuning with Vector Search

Learn how to use vector search to create highly targeted training for any classification model using a final linear classification layer. Easily fine-tune models in 10 minutes with less than 100 labeled examples.

🌲 Pinecone article:
https://pinecone.io/learn/classifier-train-vector-search/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
01:14 Classification
02:49 Better Classifier Training
06:33 Classification as Vector Search
08:47 How Fine-tuning Works
10:50 Identifying Important Samples
12:39 CODE IMPLEMENTATION
13:13 Indexing
18:59 Fine-tuning the Classifier
27:37 Classifier Predictions
30:43 Closing Notes


Hugging Face Datasets #1 - Hosting your datasets

Introduction to Hugging Face datasets, how it works, and how to host your own simple datasets (JSONL, TSV, CSV, etc) for free via Hugging Face Datasets Hub

Warp download:
https://app.warp.dev/referral/7G3N39

Git LFS Install:
Mac:
$ brew install git-lfs
Debian/Ubuntu:
$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
$ sudo apt-get install git-lfs
Windows:
Get install from https://github.com/git-lfs/git-lfs/releases

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
04:36 Creating our own Datasets
08:29 Creating JSONL for Hugging Face
15:15 Uploading Datasets for Git
19:10 LFS for Large Files
21:56 Closing Notes


CLIP Explained | Multi-modal ML

Language models (LMs) can not rely on language alone. That is the idea behind the "Experience Grounds Language" paper, that proposes a framework to measure LMs' current and future progress. A key idea is that, beyond a certain threshold LMs need other forms of data, such as visual input.

The next step beyond well-known language models; BERT, GPT-3, and T5 is ”World Scope 3”. In World Scope 3, we move from large text-only datasets to large multi-modal datasets. That is, datasets containing information from multiple forms of media, like *both* images and text.

The world, both digital and real, is multi-modal. We perceive the world as an orchestra of language, imagery, video, smell, touch, and more. This chaotic ensemble produces an inner state, our "model" of the outside world.

AI must move in the same direction. Even specialist models that focus on language or vision must, at some point, have input from the other modalities. How can a model fully understand the concept of the word "person" without *seeing* a person?

OpenAI's Contrastive Learning In Pretraining (CLIP) is a world scope three model. It can comprehend concepts in both text and image and even connect concepts between the two modalities. In this video we will learn about multi-modality, how CLIP works, and how to use CLIP for different use cases like encoding, classification, and object detection.

🌲 Pinecone article:
https://pinecone.io/learn/clip/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Hugging Face Datasets #2 - Dataset Builder Scripts

How to work with dataset builder scripts, intro to the download manager, and Apache Arrow datatypes used in Hugging Face Datasets.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

00:00 Intro
00:49 Creating Compressed Files
02:41 Creating Dataset Build Script
04:49 Download Manager
08:59 Finishing Split Generator
10:13 Generate Examples Method
14:47 Add Dataset to Hugging Face
17:49 Apache Arrow Features
22:52 What's Next?


Zero-Shot Image Classification with OpenAI's CLIP

State-of-the-art (SotA) computer vision (CV) models are characterized by a *restricted* understanding of the visual world specific to their training data [1].

These models can perform *very well* on specific tasks and datasets, but they do not generalize well. They cannot handle new classes or images beyond the domain they have been trained with.

Ideally, a CV model should learn the contents of images without excessive focus on the specific labels it is initially trained to understand.

Fortunately, OpenAI's CLIP has proved itself as an incredibly flexible CV classification model that often requires *zero* retraining. In this chapter, we will explore CLIP in zero-shot image classification.

🌲 Pinecone article:
https://pinecone.io/learn/clip-classification/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Subscribe for Article and Video Updates!
https://jamescalam.medium.com/subscribe
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP


Stoic Philosophy Text Generation with TensorFlow

Explanation of key parts to a RNN text generator built in TensorFlow with Python.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

I've written a couple of Medium articles on this project, if you're interested check them out here:
Stoic Philosophy - Built by Algorithms
https://towardsdatascience.com/stoic-philosophy-built-by-algorithms-9cff7b91dcbd
Supercharged Prediction with Ensemble Learning
https://towardsdatascience.com/recurrent-ensemble-learning-caffdcd94092

Music used by Lakey Inspired.
1 - Blue Boi
2 - Falling
https://www.youtube.com/channel/UCOmy8wuTpC95lefU5d1dt2Q


How to Build TensorFlow Pipelines with tf.data.Dataset

Link to updated version (without video freeze): https://youtu.be/f6XVfgJTbp4

An introduction to building better input pipelines for Machine Learning in TF2.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Link to tf.data API docs: https://www.tensorflow.org/guide/data


Every New Feature in Python 3.10.0a2

Every new feature in the early release alpha 2 preview of Python 3.10

There is video lag 5:00 - 9:55 covering the Type Alias section (sorry!) - the audio is okay though

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


How-to Build a Transformer for Language Classification in TensorFlow

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

How to build a transformer model for sentiment analysis (language classification) using HuggingFace's Transformers library in TensorFlow 2 with Python.

We cover the full process from downloading data all the way through to building and training the transformer model.

This is a multi-class classification problem using both TensorFlow and Transformers to build a multiclass sentiment classifier.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Article version is here:
https://betterprogramming.pub/build-a-natural-language-classifier-with-bert-and-tensorflow-4770d4442d41

Or here (free link if you don't have Medium membership):
https://betterprogramming.pub/build-a-natural-language-classifier-with-bert-and-tensorflow-4770d4442d41?sk=346cd4ce5ee019c400835588b56d8574

Article extract:
"High-performance transformer models like BERT and GPT-3 are transforming a huge array of previously menial, language-based tasks, into the work of a few clicks, saving a lot of time.

In most industries, the newest wave of language optimization is just getting started — taking their first baby steps. But these seedlings are widespread, and sprouting quickly.

Much of this adoption is thanks to the incredibly low barrier-to-entry. If you know the basics of TensorFlow or PyTorch, and take a little time to get to grips with the Transformers library — you’re already halfway there.

With the Transformers library, it takes just three lines of code to initialize a cutting-edge ML model — a model built from the billions of research dollars spent by the likes of Google, Facebook, and OpenAI.

This article will take you through the steps to build a classification model that leverages the power of transformers, using Google’s BERT.

Transformers
- Finding Models
- Initializing
- Bert Inputs and Outputs
Classification
- The Data
- Tokenization
- Data Prep
- Train-Validation Split
- Model Definition
- Train"


How-to use the Kaggle API in Python

Simple step-by-step tutorial covering the setup and use of the Kaggle API for downloading datasets using the Kaggle library in Python.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


Language Generation with OpenAI's GPT-2 in Python

Easy natural language generation with Transformers and PyTorch. We apply OpenAI's GPT-2 model to generate text in just a few lines of Python code.

Language generation is one of those natural language tasks that can really produce an incredible feeling of awe at how far the fields of machine learning and artificial intelligence have come.

GPT-1, 2, and 3 are OpenAI’s top language models — well known for their ability to produce incredibly natural, coherent, and genuinely interesting language.

In this article, we will take a small snippet of text and learn how to feed that into a pre-trained GPT-2 model using PyTorch and Transformers to produce high-quality language generation in just eight lines of code. We cover:

PyTorch and Transformers
- Data
Building the Model
- Initialization
- Tokenization
- Generation
- Decoding
Results

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium Article:
https://towardsdatascience.com/text-generation-with-python-and-gpt-2-1fecbff1635b

Friend Link (free access):
https://towardsdatascience.com/text-generation-with-python-and-gpt-2-1fecbff1635b?sk=930367d835f15abb4ef3164f7791e1b1

Thumbnail background by gustavo centurion on Unsplash
https://unsplash.com/photos/O6fs4ablxw8


Text Summarization with Google AI's T5 in Python

Easy text summarization using Google AI's T5 model using HuggingFace transformers and PyTorch in Python.

Automatic text summarization allows us to shorten long pieces of text into easy-to-read, short snippets that still convey the most important and relevant information of the original text.

In this video, we’ll build a simple but incredibly powerful text summarizer using Google’s T5. We’ll be using the PyTorch and HuggingFace’s Transformers frameworks.

This is split into three parts:
1. Import and Initialization
2. Data and Tokenization
3. Summary Generation

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

You can read the article version of this on Medium here:
https://betterprogramming.pub/how-to-summarize-text-with-googles-t5-4dd1ae6238b6

(And for those of you without Medium membership, here's a free link):
https://betterprogramming.pub/how-to-summarize-text-with-googles-t5-4dd1ae6238b6?sk=740d3009282cb2c4f7478a0c073dedb3

Thumbnail background by gustavo centurion on Unsplash
https://unsplash.com/photos/O6fs4ablxw8


How-to do Sentiment Analysis with Flair in Python

Learn how to perform powerful sentiment analysis with no fine-tuning or pre-training required using the Flair NLP library in Python.

With the real-time information available to us on massive social media platforms like Twitter, we have all the data we could ever need to create these accurate and up-to-date sentiment metrics for different companies.

But then comes the question, how can our computer understand what this unstructured text data means?

That is where sentiment analysis comes in. Sentiment analysis is a particularly interesting branch of Natural Language Processing (NLP), which is used to rate the language used in a body of text.

Through sentiment analysis, we can take thousands of tweets about a company and judge whether they are generally positive or negative (the sentiment) in real-time!

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/sentiment-analysis-for-stock-price-prediction-in-python-bed40c65d178

(Free link if you don't have Medium membership):
https://towardsdatascience.com/sentiment-analysis-for-stock-price-prediction-in-python-bed40c65d178?sk=1cbf33a5d1fd2ed841f9487972c1cbed

Thumbnail photo by Alexander London on Unsplash
https://unsplash.com/@alxndr_london


Python Environment Setup for Machine Learning

Everything you need for a Python environment set up for Machine Learning and Data Science!

📕 Article:
https://towardsdatascience.com/how-to-setup-python-for-machine-learning-173cb25f0206

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Thumbnail background by Christian Wiediger on Unsplash
https://unsplash.com/@christianw


Functional API - TensorFlow Essentials #2

A look at the functional API method for building models in TensorFlow 2 for Python.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Thumbnail background by Darius Bashar on Unsplash
https://unsplash.com/@dariusbashar?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText


Training Parameters - TensorFlow Essentials #3

Learn how to set up model training parameters and compile the model before training.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Thumbnail background by Alex McCarthy on Unsplash
https://unsplash.com/@4lexmccarthy?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText


Input Data Pipelines - TensorFlow Essentials #4

Learn how to set-up efficient and clean input data pipelines using tf.data.Dataset

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Thumbnail background by Daria Nepriakhina on Unsplash
https://unsplash.com/@epicantus?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText


Sequential Model - TensorFlow Essentials #1

Learn how to use the sequential model building approach in TensorFlow 2.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Background thumbnail by Aryan Dhiman on Unsplash
https://unsplash.com/@mylifeasaryan_?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText


6 of Python's Newest and Best Features (3.7-3.9)

A rundown of the six most recent, and coolest features added to Python in the past few years!

2018 brought us a plethora of new features with the release of Python 3.7, followed by 3.8 in 2019, and 3.9 in 2020.

Many of those changes were behind the scenes. Optimizations and upgrades that the vast majority of us will never notice, despite their benefits.

Others are more obvious, additions to syntax or functionality that can change how we write our code. But even these visible changes can be hard to keep up with.

In this video, we will run through the more apparent upgrades to provide a brief but hopefully invaluable refresher on everything new to Python from the past few years.

- Python 3.7
  - Breakpoints
- Python 3.8
  - Walrus Operator
  - F-string '=' Specifier
  - Positional-only Parameters
- Python 3.9
  - More Type Hinting
  - Dictionary Unions

Medium Article:
https://towardsdatascience.com/amazing-features-added-to-python-from-3-7-to-now-4f35f0bb1ea6

(Free access link):
https://towardsdatascience.com/amazing-features-added-to-python-from-3-7-to-now-4f35f0bb1ea6?sk=bda3cb7717caa969b81619f85191f241

Thumbnail background by Martin Sanchez on Unsplash:
https://unsplash.com/photos/4PDPLw1flgE

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


Novice to Advanced RegEx in Less-than 30 Minutes + Python

A full tutorial covering everything you need to know about Regular Expressions - an essential for anyone learning to code - and even more so for anyone interested in Natural Language Processing.

This video includes:

- metacharacters
- quantifiers
- capture groups
- using capture groups in Python
- character sets
- look-ahead and look-behind assertions
- negative look-ahead and look-behind assertions
- inline modifiers
- passing modifiers as function parameters in Python
- conditionals (if-else statements for RegEx)
- re.match
- re.search
- re.findall

We cover all of this in-depth in this tutorial, incl. examples all the way through on RegEx101 (an interactive debugging/regex building tool) and also in Python.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


Building a PlotLy $GME Chart in Python

A code-along video covering the coding process from imagination to Python.
Something a little different, I'm not overly keen on this format - it's pretty long - but I've recorded it and I think maybe this can be useful for a few of you.
I haven't prepared anything beforehand, this is just going into the coding process with a rough outline of wanting to build a stock chart for GME (GameStop) and adding a few technical indicators - to get more familiar with PlotLy and the AlphaVantage API.
So, it's a weird one, but I hope a few of you enjoy it - thanks :)

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


How to Build Custom Q&A Transformer Models in Python

In this video, we will learn how to take a pre-trained transformer model and train it for question-and-answering. We will be using the HuggingFace transformers library with the PyTorch implementation of models in Python.

Transformers are one of the biggest developments in Natural Language Processing (NLP) and learning how to use them properly is basically a data science superpower - they're genuinely amazing I promise!

I hope you enjoy the video :)

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/the-ultimate-performance-metric-in-nlp-111df6c64460

(Free link):
https://towardsdatascience.com/how-to-fine-tune-a-q-a-transformer-86f91ec92997?sk=9344fd51afe71a0905db833d0183d436

Code:
https://gist.github.com/jamescalam/55daf50c8da9eb3a7c18de058bc139a3

Photo in thumbnail by Lorenzo Herrera on Unsplash
https://unsplash.com/@lorenzoherrera


How-to Use The Reddit API in Python

Learn how to use the Reddit API in Python, including setup, authorization, and pulling data from subreddits.

Reddit API docs:
https://www.reddit.com/dev/api/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/how-to-use-the-reddit-api-in-python-5e05ddfd1e5c

📖 Free link:
https://towardsdatascience.com/how-to-use-the-reddit-api-in-python-5e05ddfd1e5c?sk=0295f297c1365bee7cc7a32bdff21b61

Extract from article:

"Reddit is a huge ecosystem brimming with data that is readily available at our very fingertips. As a data-minded person, I wanted to take advantage of this and perform some analysis using this vast repository of open-source data.
Initially, it turned out that getting to grip with Reddit’s API wasn’t as clear-cut as expected — despite being a straightforward process; it can be a little confusing at first.
So, after figuring everything out, I wrote this article — which I hope will help a few of you to get familiar with using the Reddit API in Python. We will cover:
Getting Access
Making Requests
  - Reading the Data
  - Streaming New Posts
Parameters

Getting Access
First, we need access. Unlike most popular services, the Reddit API was somewhat difficult to figure out initially. There are several steps:
1. Go to App Preferences and click create another app… at the bottom.
2. Fill out the required details, make sure to select script — and click create app.
3. make a note of the personal use script and secret tokens.
4. Request a temporary OAuth token from Reddit. We need our username and password for this.
5. Add headers=headers to every request. The OAuth token will expire after ~2 hours, and a new one will need to be requested.
"

And so on, check it out if you're interested in reading (rather than watching).

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


How to Build Q&A Models in Python (Transformers)

In this video we'll cover how to build a question-answering model in Python using HuggingFace's Transformers.

You will need to install the transformers library with:
pip install transformers

Alongside either TensorFlow or PyTorch (to follow this video exactly you will need PyTorch). To install TensorFlow just type:
pip install tensorflow
OR
conda install tensorflow

And for PyTorch follow the instructions under 'Install PyTorch' here:
https://pytorch.org/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Link to Q&A fine-tuning video:
https://youtu.be/ZIRmXkHp0-c

You can find the Medium article link below here:
https://towardsdatascience.com/question-and-answering-with-bert-6ef89a78dac


How-to Decode Outputs From NLP Models (Python)

In this video, we will cover three ways to decode the output probabilities from NLP models - greedy search, random sampling, and beam search.

Learning how to decode outputs can make a huge difference in diagnosing model issues and improving text output quality - and as an added bonus it's super easy.

One of the often-overlooked parts of sequence generation in natural language processing (NLP) is how we select our output tokens — otherwise known as decoding.

You may be thinking — we select a token/word/character based on the probability of each token assigned by our model.

This is half-true — in language-based tasks, we typically build a model which outputs a set of probabilities to an array where each value in that array represents the probability of a specific word/token.

At this point, it might seem logical to select the token with the highest probability? Well, not really — this can create some unforeseen consequences — as we will see soon.

When we are selecting a token in machine-generated text, we have a few alternative methods for performing this decode — and options for modifying the exact behavior too.

In this video we will explore three different methods for selecting our output token, these are:

- Greedy Decoding
- Random Sampling
- Beam Search

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Link to the article version on Medium:
https://towardsdatascience.com/the-three-decoding-methods-for-nlp-23ca59cb1e9d

Free link (if you don't have membership):
https://towardsdatascience.com/the-three-decoding-methods-for-nlp-23ca59cb1e9d?sk=64fbb0204c174dc520af027a69f88030


Identify Stocks on Reddit with SpaCy (NER in Python)

We will learn how to process unstructured text data from Reddit and extract organization names so that any further analysis is automatically classified and results assigned to the correct stocks.

Organizations are mentioned in each subreddit in a variety of formats. Typically we will find two formats:

- Organization name, eg Tesla/Tesla Motors
- Ticker symbol, eg TSLA, tsla, or $TSLA

We also need to be able to differentiate between tickers and other abbreviations/slang -some of these are unclear like AI (AI can mean both artificial intelligence and refer to the ticker symbol for C3.ai).

So, we need a reasonable competent NER process to accurately classify our data.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Reddit API video: https://youtu.be/FdjVoOf9HN4
/r/investing data: https://github.com/jamescalam/transformers/blob/main/course/named_entity_recognition/data/reddit_investing.csv
Medium article: https://towardsdatascience.com/ner-for-extracting-stock-mentions-on-reddit-aa604e577be
(Free version if you don't have Medium membership): https://towardsdatascience.com/ner-for-extracting-stock-mentions-on-reddit-aa604e577be?sk=d16305d40b18e7955a0665633182d2b4

Thanks for watching!


Sentiment Analysis on ANY Length of Text With Transformers (Python)

The de-facto standard in many natural language processing (NLP) tasks nowadays is to use a transformer. Text generation? Transformer. Question-and-answering? Transformer. Language classification? Transformer!

However, one of the problems with many of these models (a problem that is not just restricted to transformer models)  is that we cannot process long pieces of text.

Almost every article I write on Medium contains 1000+ words, which, when tokenized for a transformer model like BERT, will produce 1000+ tokens. BERT (and many other transformer models) will consume 512 tokens max - truncating anything beyond this length.

Although I think you may struggle to find value in processing my Medium articles, the same applies to many useful data sources - like news articles or Reddit posts.

We will take a look at how we can work around this limitation. In this article, we will find the sentiment for long posts from the /r/investing subreddit. This video will cover:

High-Level Approach
Getting Started
- Data
- Initialization
Tokenization
Preparing The Chunks
- Split
- CLS and SEP
- Padding
- Reshaping For BERT
Making Predictions

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Here's a link to the Medium article:
https://towardsdatascience.com/how-to-apply-transformers-to-any-length-of-text-a5601410af7f

And a free access link if you don't have Medium membership:
https://towardsdatascience.com/how-to-apply-transformers-to-any-length-of-text-a5601410af7f?sk=d4e717eb2ff31fb27ea67019bbb63ad6


Unicode Normalization for NLP in Python

ℕ𝕠-𝕠𝕟𝕖 𝕚𝕟 𝕥𝕙𝕖𝕚𝕣 𝕣𝕚𝕘𝕙𝕥 𝕞𝕚𝕟𝕕 𝕨𝕠𝕦𝕝𝕕 𝕖𝕧𝕖𝕣 𝕦𝕤𝕖 𝕥𝕙𝕖𝕤𝕖 𝕒𝕟𝕟𝕠𝕪𝕚𝕟𝕘 𝕗𝕠𝕟𝕥 𝕧𝕒𝕣𝕚𝕒𝕟𝕥𝕤. 𝕋𝕙𝕖 𝕨𝕠𝕣𝕤𝕥 𝕥𝕙𝕚𝕟𝕘, 𝕚𝕤 𝕚𝕗 𝕪𝕠𝕦 𝕕𝕠 𝕒𝕟𝕪 𝕗𝕠𝕣𝕞 𝕠𝕗 ℕ𝕃ℙ 𝕒𝕟𝕕 𝕪𝕠𝕦 𝕙𝕒𝕧𝕖 𝕔𝕙𝕒𝕣𝕒𝕔𝕥𝕖𝕣𝕤 𝕝𝕚𝕜𝕖 𝕥𝕙𝕚𝕤 𝕚𝕟 𝕪𝕠𝕦𝕣 𝕚𝕟𝕡𝕦𝕥, 𝕪𝕠𝕦𝕣 𝕥𝕖𝕩𝕥 𝕓𝕖𝕔𝕠𝕞𝕖𝕤 𝕔𝕠𝕞𝕡𝕝𝕖𝕥𝕖𝕝𝕪 𝕦𝕟𝕣𝕖𝕒𝕕𝕒𝕓𝕝𝕖.

We also find that text like this is incredibly common - particularly on social media.

Another pain-point comes from diacritics (the little glyphs in Ç, é, Å) that you'll find in almost every European language.

These characters have a hidden property that can trip up any NLP model - take a look at the Unicode for two versions of Ç:

Latin capital letter C with cedilla: \u00C7

Latin capital letter C + combining cedilla: \u0043\u0327

Both are completely different, despite rendering as the same character.

To deal with all of these text variants we need to use Unicode normalization - which we will cover in this video.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/what-on-earth-is-unicode-normalization-56c005c55ad0

Friend link (free access):
https://towardsdatascience.com/what-on-earth-is-unicode-normalization-56c005c55ad0?sk=0cd19a9ad9f5d948b33179bab3c3b7cd


The NEW Match-Case Statement in Python 3.10

Python 3.10 is beginning to fill-out with plenty of fascinating new features. One of those, in particular, caught my attention - structural pattern matching - or as most of us will know it, switch/case statements.

Switch-statements have been absent from Python despite being a common feature of most languages. Python is leapfrogging ahead of those languages by introducing the match-case statement as a switch-case v2.0.

Back in 2006, PEP 3103 was raised, recommending the implementation of a switch-case statement. However, after a poll at PyCon 2007 received no support for the feature, the Python devs dropped it.

Fast-forward to 2020, and Guido van Rossum, the creator of Python, committed the first documentation showing the new match-statements, which have been named Structural Pattern Matching, as found in PEP 634.

Let's take a look at how this new logic works.

Medium Article:
https://towardsdatascience.com/switch-case-statements-are-coming-to-python-d0caf7b2bfd3

Friend Link (free access):
https://towardsdatascience.com/switch-case-statements-are-coming-to-python-d0caf7b2bfd3?sk=363e0f7696502647e007f91910b4c817

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:58 Switch-Case
02:37 Flow of Logic
03:21 Second Example (Tuples)
05:00 Final Example Setup
11:30 Final Example If-Else Version
15:22 Final Example Match-Case Version


Multi-Class Language Classification With BERT in TensorFlow

Chapters for each section of the video (preprocessing, model build, prediction) are in the video timeline.

Transformers have been described as the fourth pillar of deep learning [1], alongside the three big neural net architectures of CNNs, RNNs, and MLPs.

However, from the perspective of natural language processing - transformers are much more than that. Since their introduction in 2017, they've come to dominate a majority of NLP benchmarks - and continue to impress daily.

What I'm saying is, transformers are damn cool. And with libraries like HuggingFace's transformers - it has become too easy to build incredible solutions with them.

So, what's not to love? Incredible performance paired with the ultimate ease-of-use.

In this video, we'll work through building a multi-class classification model using transformers - from start-to-finish.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/multi-class-classification-with-transformers-6cf7b59a033a

Free access:
https://towardsdatascience.com/multi-class-classification-with-transformers-6cf7b59a033a?sk=544872025c2283c54cf4294814b8cae3

Link to Kaggle video:
https://youtu.be/DgGFhQmfxHo

[1] Fourth Pillar of AI:
https://ark-invest.com/articles/analyst-research/transformers-comprise-the-fourth-pillar-of-deep-learning/

00:00 Intro
01:21 Pulling Data
01:47 Preprocessing
14:33 Data Input Pipeline
24:14 Defining Model
33:29 Model Training
35:36 Saving and Loading Models
37:37 Making Predictions


How to Build Python Packages for Pip

The most powerful feature of Python is its community. Almost every use-case out there has a package built specifically for it.

Need to send mobile/email alerts? pip install knockknock  -  Build ML apps? pip install streamlit  -  Bored of your terminal? pip install colorama  -  It's too easy!

I know this is obvious, but those libraries didn't magically appear. For each package, there is a person, or many persons - that actively developed and deployed that package.

Every single one.

All 300K+ of them.

That is why Python is Python, the level of support is phenomenal - mindblowing.

In this video, we will learn how to build our own packages. And add them to the Python Package Index (PyPI). Afterward, we will be able to install our packages using pip install!

GitHub Repo:
https://github.com/jamescalam/aesthetic_ascii

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium Article:
https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e

📖 Here's a free link:
https://towardsdatascience.com/how-to-package-your-python-code-df5a7739ab2e?sk=04d9f67c0654445bbcbbf6825f535900


How-to Structure a Q&A ML App

▶ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

I'm planning on doing something different, a series of videos where we work through the steps - from start-to-finish - of (attempting) to build a Q&A web app that answers our questions with Stoic answers.

In this video, I'm outlining the idea and describing the high-level setup that I think we'll need to put together. It should be cool!

We'll be using the Haystack framework for 'Q&A at scale', which using HuggingFace transformers under-the-hood, and the Elasticsearch document store.

Find the repo here:
https://github.com/jamescalam/aurelius

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


How to Index Q&A Data With Haystack and Elasticsearch

▶ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

The second video in 'Building a Stoic Q&A App' - here we're setting up Elasticsearch and Haystack to store the data (Meditations) ready for retrieval when we ask our app questions.

Find the code here:
https://github.com/jamescalam/aurelius/tree/main/code/labs

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


Q&A Document Retrieval With DPR

▶ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

The third video in building our Stoic Q&A app.

In open-domain question answering, we typically design a model architecture that contains a data source, retriever, and reader/generator.

The first of these components is typically a document store. The two most popular stores we use here are Elasticsearch and FAISS.

Next up is our retriever — the topic of this video. The job of the retriever is to filter through our document store for relevant chunks of information (the documents) and pass them to the reader/generator model.

DPR (dense passage retriever) is a dense vector retriever that is trained on question-context pairs. Encoding both accordingly - enabling super accurate similarity indexing.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

If you're interested in learning more about DPR, I wrote about it on Medium here:
https://towardsdatascience.com/how-to-create-an-answer-from-a-question-with-dpr-d76e29cc5d60

(Free link):
https://towardsdatascience.com/how-to-create-an-answer-from-a-question-with-dpr-d76e29cc5d60?sk=1bdd7c1bff80bf51410962691c690c69

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


How to Use Type Annotations in Python

Type annotations - also known as type signatures - are used to indicate the datatypes of variables and input/outputs of functions and methods.

In many languages, datatypes are explicitly stated. In these languages, if you don't declare your datatype - the code will not run.

Type annotations have a long and convoluted history with Python, going all the way back to the first release of Python 3 with the initial implementation of function annotations.

Type annotations in Python are not make-or-break like in other languages (like C). They're optional chunks of syntax that we can add to make our code more explicit.

Erroneous type annotations will do nothing more than highlight the incorrect annotation in our code editor - no errors are ever raised due to annotations.

So, if type annotations are not enforced, why use them?

Well, as we touched upon already - declaring types makes our code more explicit, and if done well, easier to read - both for ourselves and others.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Read the Medium article here:
https://towardsdatascience.com/type-annotations-in-python-d90990b172dc

📖 Here's a free link:
https://towardsdatascience.com/type-annotations-in-python-d90990b172dc?sk=29bc29ab5478a842363963b421781b47

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:55 Datatypes Example in C
2:53 Static and Dynamic Typed Languages
3:47 Type Annotations in Python
4:25 How to Define Simple Types
6:04 IDE Warnings
8:20 More Complex Types
9:53 dict[str, int]
11.07 Multiple Types
11:38 Union Operator (Py 3.9)
12:34 Union Operator (Py 3.10)
13:21 Optional Operator


Extractive Q&A With Haystack and FastAPI in Python

▶ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

In this video we work through building an extractive Q&A stack using Haystack, and embedding it within a FastAPI instance in Python.

We use the BERT transformer for our reader model, alongside Elasticsearch and the BM25 retriever algorithm.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Sentence Similarity With Transformers and PyTorch (Python)

Easy mode: https://youtu.be/Ey81KfQ3PQU

All we ever seem to talk about nowadays are BERT this, BERT that. I want to talk about something else, but BERT is just too good  -  so this video will be about BERT for sentence similarity.

A big part of NLP relies on similarity in highly-dimensional spaces. Typically an NLP solution will take some text, process it to create a big vector/array representing said text - then perform several transformations.

It's highly-dimensional magic.

Sentence similarity is one of the clearest examples of how powerful highly-dimensional magic can be.

The logic is this:
- Take a sentence, convert it into a vector.
- Take many other sentences, and convert them into vectors.
- Find sentences that have the smallest distance (Euclidean) or smallest angle (cosine similarity) between them - more on that here.
- We now have a measure of semantic similarity between sentences - easy!

At a high level, there's not much else to it. But of course, we want to understand what is happening in a little more detail and implement this in Python too.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/bert-for-measuring-text-similarity-eec91c6bf9e1

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/bert-for-measuring-text-similarity-eec91c6bf9e1?sk=c0f2990b4660210b447e52d55bd0f4e5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:16 BERT Base Network
1:11 Sentence Vectors and Similarity
1:47 The Data and Model
3:01 Two Approaches
3:16 Tokenizing Sentences
9:11 Creating last_hidden_state Tensor
11:08 Creating Sentence Vectors
17:53 Cosine Similarity


Sentence Similarity With Sentence-Transformers in Python

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

Hard mode: https://youtu.be/jVPd7lEvjtg

All we ever seem to talk about nowadays are BERT this, BERT that. I want to talk about something else, but BERT is just too good  -  so this video will be about BERT for sentence similarity.

A big part of NLP relies on similarity in highly-dimensional spaces. Typically an NLP solution will take some text, process it to create a big vector/array representing said text - then perform several transformations.

It's highly-dimensional magic.

Sentence similarity is one of the clearest examples of how powerful highly-dimensional magic can be.

The logic is this:
- Take a sentence, convert it into a vector.
- Take many other sentences, and convert them into vectors.
- Find sentences that have the smallest distance (Euclidean) or smallest angle (cosine similarity) between them - more on that here.
- We now have a measure of semantic similarity between sentences - easy!

At a high level, there's not much else to it. But of course, we want to understand what is happening in a little more detail and implement this in Python too.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/bert-for-measuring-text-similarity-eec91c6bf9e1

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/bert-for-measuring-text-similarity-eec91c6bf9e1?sk=c0f2990b4660210b447e52d55bd0f4e5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


NER With Transformers and spaCy (Python)

Named entity recognition (NER) consists of extracting 'entities' from text - what we mean by that is given the sentence:

"Apple reached an all-time high stock price of 143 dollars this January."

We might want to extract the key pieces of information - or 'entities' - and categorize each of those entities. Like so:

- Apple  : Organization
- 143 dollars :  Monetary Value
- this January :  Date

For us humans, this is easy. But how can we teach a machine to distinguish between a granny smith apple and the Apple we trade on NASDAQ?

(No, we can't rely on the 'A' being capitalized…)

This is where NER comes in - using NER, we can extract keywords like apple and identify that it is, in fact, an organization - not a fruit.

The go-to library for NER is spaCy, which is incredible. But what if we added transformers to spaCy? Even better - we'll cover exactly that in this video.

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5


Training BERT #1 - Masked-Language Modeling (MLM)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

BERT, everyone's favorite transformer costs Google ~$7K to train (and who knows how much in R&D costs). From there, we write a couple of lines of code to use the same model - all for free.

BERT has enjoyed unparalleled success in NLP thanks to two unique training approaches, masked-language modeling (MLM), and next sentence prediction (NSP).

MLM consists of giving BERT a sentence and optimizing the weights inside BERT to output the same sentence on the other side.

So we input a sentence and ask that BERT outputs the same sentence.

However, before we actually give BERT that input sentence - we mask a few tokens.

So we're actually inputting an incomplete sentence and asking BERT to complete it for us.

How to train BERT with MLM:
https://youtu.be/R6hcxMMOrPE

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

Medium article:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c?sk=17a19eca8dc8280bea4138802580ffe0

🤖 70% Discount on the NLP With Transformers in Python course:
https://www.udemy.com/course/nlp-with-transformers/?couponCode=MEDIUM3

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Training BERT #2 - Train With Masked-Language Modeling (MLM)

🎁 Free NLP for Semantic Search Course:
https://www.pinecone.io/learn/nlp

BERT has enjoyed unparalleled success in NLP thanks to two unique training approaches, masked-language modeling (MLM), and next sentence prediction (NSP).

In many cases, we might be able to take the pre-trained BERT model out-of-the-box and apply it successfully to our own language tasks.

But often, we might need to pre-train the model for a specific use case even further.

Further training with MLM allows us to tune BERT to better understand the particular use of language in a more specific domain.

Out-of-the-box BERT - great for general purpose use. Fine-tuned with MLM BERT - great for domain-specific use.

In this video, we'll cover exactly how to fine-tune BERT models using MLM in PyTorch.

👾 Code:
https://github.com/jamescalam/transformers/blob/main/course/training/03_mlm_training.ipynb

Meditations data:
https://github.com/jamescalam/transformers/blob/main/data/text/meditations/clean.txt

Understanding MLM:
https://youtu.be/q9NS5WpfkrU

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/masked-language-modelling-with-bert-7d49793e5d2c?sk=17a19eca8dc8280bea4138802580ffe0

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Training BERT #3 - Next Sentence Prediction (NSP)

Next sentence prediction (NSP) is one-half of the training process behind the BERT model (the other being masked-language modeling - MLM).

Where MLM teaches BERT to understand relationships between words - NSP teaches BERT to understand relationships between sentences.

In the original BERT paper, it was found that without NSP, BERT performed worse on every single metric -  so it's important.

Now, when we use a pre-trained BERT model, training with NSP and MLM has already been done, so why do we need to know about it?

Well, we can actually further pre-train these pre-trained BERT models so that they better understand the language used in our specific use-cases. To do that, we can use both MLM and NSP.

So, in this video, we'll go into depth on what NSP is, how it works, and how we can implement it in code.

Training with NSP:
https://youtu.be/x1lAcT3xl5M

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/bert-for-next-sentence-prediction-466b67f8226f

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/bert-for-next-sentence-prediction-466b67f8226f?sk=3595968413abde1c5833e1a96e449673

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Training BERT #4 - Train With Next Sentence Prediction (NSP)

Next sentence prediction (NSP) is one-half of the training process behind the BERT model (the other being masked-language modeling - MLM).

Although NSP (and MLM) are used to pre-train BERT models, we can use these exact methods to further pre-train our models to better understand the specific style of language in our own use cases.

So, in this video, we'll cover exactly how we take an unstructured body of text, and use it to pre-train a BERT model using NSP.

Meditations data:
https://github.com/jamescalam/transformers/blob/main/data/text/meditations/clean.txt

Jupyter Notebook
https://github.com/jamescalam/transformers/blob/main/course/training/06_nsp_training.ipynb

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/bert-for-next-sentence-prediction-466b67f8226f

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/bert-for-next-sentence-prediction-466b67f8226f?sk=3595968413abde1c5833e1a96e449673

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


New Features in Python 3.10

The Python 3.10 release has several new features like structural pattern matching, a new typing Union operator, and parenthesized context managers!

Python 3.10 has now been released, here we test all of the best new features introduced.

We'll cover some of the most interesting additions to Python - structural pattern matching, parenthesized context managers, more typing, and the new and improved error messages.

Download the latest release:
https://www.python.org/downloads/release/python-3100/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/whats-new-in-python-3-10-a757c6c69342

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/whats-new-in-python-3-10-a757c6c69342?sk=648ae12c1025a83affba4eecec0d46c6

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:45 Type Annotations in Python
01:10  Typing Union Operator
02:07 Parenthesized Context Managers
05:07 Structural Pattern Matching
09:31 Better Error Messages


Training BERT #5 - Training With BertForPretraining

NSP Logic
https://youtu.be/1gN1snKBLP0

MLM Logic
https://youtu.be/q9NS5WpfkrU

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/how-to-train-bert-aaad00533168

📖 Here's a free link:
https://towardsdatascience.com/how-to-train-bert-aaad00533168?sk=5ad4e5e44a6c573b3be1967c9abdcc35

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


FREE 11 Hour NLP Transformers Course (Next 3 Days Only)

The offer has now expired! You can find the final 70% discount here:
https://bit.ly/3DFvvY5

In total, 10823 people redeemed the code - which is incredible, I'm very happy so many of you were interested in the course and I hope it will help many of you in learning about transformers and NLP where it may have been too expensive to otherwise - so thank you all!

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


How-to Use HuggingFace's Datasets - Transformers From Scratch #1

How can we build our own custom transformer models?

Maybe we'd like our model to understand a less common language, how many transformer models out there have been trained on Piemontese or the Nahuatl languages?

In that case, we need to do something different. We need to build our own model - from scratch.

In this video, we'll learn how to use HuggingFace's datasets library to download multilingual data and prepare it for training our custom transformer tokenizer and model.

---
Part 2: https://youtu.be/JIeAB8vvBQo
Part 3: https://youtu.be/heTYbpr9mD8
Part 4: https://youtu.be/35Pdoyi6ZoQ

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/transformers-from-scratch-creating-a-tokenizer-7d7418adb403

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/transformers-from-scratch-creating-a-tokenizer-7d7418adb403?sk=aea909609f41be43bdb2dbbd75a801f2

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Build a Custom Transformer Tokenizer - Transformers From Scratch #2

How can we build our own custom transformer models?

Maybe we'd like our model to understand a less common language, how many transformer models out there have been trained on Piemontese or the Nahuatl languages?

In that case, we need to do something different. We need to build our own model - from scratch.

In this video, we'll learn how to use HuggingFace's tokenizers library to build our own custom transformer tokenizer.

Part 1: https://youtu.be/GhGUZrcB-WM
---
Part 3: https://youtu.be/heTYbpr9mD8
Part 4: https://youtu.be/35Pdoyi6ZoQ

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

📙 Medium article:
https://towardsdatascience.com/transformers-from-scratch-creating-a-tokenizer-7d7418adb403

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/transformers-from-scratch-creating-a-tokenizer-7d7418adb403?sk=aea909609f41be43bdb2dbbd75a801f2

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


3 Vector-based Methods for Similarity Search (TF-IDF, BM25, SBERT)

Vector similarity search is one of the fastest-growing domains in AI and machine learning. At its core, it is the process of matching relevant pieces of information together.

Similarity search is a complex topic and there are countless techniques for building effective search engines.

In this video, we'll cover three vector-based approaches for comparing languages and identifying similar 'documents', covering both vector similarity search and semantic search:

- TF-IDF
- BM25
- Sentence-BERT

📰 Original article:
https://www.pinecone.io/learn/semantic-search/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
01:37 TF-IDF
11:44 BM25
20:30 SBERT


3 Traditional Methods for Similarity Search (Jaccard, w-shingling, Levenshtein)

Similarity search is one of the fastest-growing domains in AI and machine learning. At its core, it is the process of matching relevant pieces of information together.

Similarity search is a complex topic and there are countless techniques for building effective search engines.

In this video, we'll cover three traditional approaches for comparing languages and identifying similar 'documents':

- Jaccard Similarity
- w-shingling
- Levenshtein distance

📰 Original article:
https://www.pinecone.io/learn/semantic-search/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:23 Jaccard Similarity
02:39 w-shingling
07:17 Levenshtein Distance


Building MLM Training Input Pipeline - Transformers From Scratch #3

The input pipeline of our training process is the more complex part of the entire transformer build. It consists of us taking our raw OSCAR training data, transforming it, and preparing it for Masked-Language Modeling (MLM). Finally, we load our data into a DataLoader ready for training!

Part 1: https://youtu.be/GhGUZrcB-WM
Part 2: https://youtu.be/JIeAB8vvBQo
---
Part 4: https://youtu.be/35Pdoyi6ZoQ

📙 Medium article:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6

📖 Free link:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6?sk=9db6224efbd4ec6fd407a80b528e69b0

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Angular App Setup With Material - Stoic Q&A #5

▶ Stoic Q&A App Playlist: https://www.youtube.com/playlist?list=PLIUOU7oqGTLixb-CatMxNCO-mJioMmZEB

The fifth video in our Stoic Q&A series - setting up our Angular app with Angular Material.

Prerequisites:
Installation of Node.js and NPM - https://nodejs.org/en/
Angular - https://angular.io/guide/setup-local

👾 Discord
https://discord.gg/c5QtDB9RAP


Training and Testing an Italian BERT - Transformers From Scratch #4

We need two things for training, our DataLoader and a model. The DataLoader we have — but no model.

For training, we need a raw (not pre-trained) RobertaForMaskedLM. To create that, we first need to create a RoBERTa config object to describe the parameters we’d like to initialize FiliBERTo with.

Once we have our model, we set up our training loop and train!

Post-training, we'll test the model with Laura, who is Italian - and hope for the best.

Part 1: https://youtu.be/GhGUZrcB-WM
Part 2: https://youtu.be/JIeAB8vvBQo
Part 3: https://youtu.be/heTYbpr9mD8
---

📙 Medium article:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/how-to-train-a-bert-model-from-scratch-72cfce554fc6?sk=9db6224efbd4ec6fd407a80b528e69b0

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
00:35 Review of Code
02:02 Config Object
06:28 Setup For Training
10:30 Training Loop
14:57 Dealing With CUDA Errors
16:17 Training Results
19:52 Loss
21:18 Fill-mask Pipeline For Testing
21:54 Testing With Laura


Faiss - Introduction to Similarity Search

Full Similarity Search Playlist:
https://www.youtube.com/watch?v=AY62z7HrghY&list=PLIUOU7oqGTLhlWpTz4NnuT3FekouIVlqc&index=1

Facebook AI Similarity Search (FAISS) is one of the most popular implementations of efficient similarity search, but what is it - and how can we use it?

What is it that makes FAISS special? How do we make the best use of this incredible tool?

Fortunately, it's a brilliantly simple process to get started with. And in this video, we'll explore some of the options FAISS provides, how they work, and - most importantly - how FAISS can make our semantic search faster.

🌲 Pinecone Article:
https://www.pinecone.io/learn/faiss-tutorial/

📊 Data:
https://github.com/jamescalam/data/tree/main/sentence_embeddings_15K

Notebook:
https://gist.github.com/jamescalam/7117aa92235a7f52141ad0654795aa48

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord
https://discord.gg/c5QtDB9RAP

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Why are there so many Tokenization methods in HF Transformers?

HuggingFace's transformers library is the de-facto standard for NLP - used by practitioners worldwide, it's powerful, flexible, and easy to use. It achieves this through a fairly large (and complex) code-base, which has resulted in the question:

"Why are there so many tokenization methods in HuggingFace transformers?"

Tokenization is the process of encoding a string of text into transformer-readable token ID integers. In this video we cover five different methods for this - do these all produce the same output, or is there a difference between them?

📙 Medium article:
https://towardsdatascience.com/why-are-there-so-many-tokenization-methods-for-transformers-a340e493b3a8

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

📖 If membership is too expensive - here's a free link:
https://towardsdatascience.com/why-are-there-so-many-tokenization-methods-for-transformers-a340e493b3a8?sk=4a7e8c88d331aef9103e153b5b799ff5

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

👾 Discord
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Choosing Indexes for Similarity Search (Faiss in Python)

Facebook AI Similarity Search (Faiss) is a game-changer in the world of search. It allows us to efficiently search a huge range of media, from GIFs to articles - with incredible accuracy in sub-second timescales for billion+ size datasets.

The success in Faiss is due to many reasons. One of those, in particular, is its flexibility. Faiss recognizes that there is no 'one-size-fits-all' in similarity search.

Instead, Faiss comes with a wide range of search indexes - which we can mix and match to our choosing.

However, this great flexibility produces a question - how do we know which size fits our use case?

Which index do we choose? Should we use multiple indexes, or is one enough?

This video will explore the pros and cons of some of the most important indexes - Flat, LSH, HNSW, and IVF. We will learn how we decide which to use and the impact of parameters in each index to build some of the best indexes for semantic search.

🌲 Pinecone Article:
https://www.pinecone.io/learn/vector-indexes/

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

Download script for Sift1M dataset:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

Similarity Search Series:
https://www.youtube.com/playlist?list=PLIUOU7oqGTLhlWpTz4NnuT3FekouIVlqc

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

👾 Discord
https://discord.gg/c5QtDB9RAP

Mining Massive Datasets Book (Similarity Search):
📚 https://amzn.to/3CC0zrc (3rd ed)
📚 https://amzn.to/3AtHSnV (1st ed, cheaper)

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


Locality Sensitive Hashing (LSH) for Search with Shingling + MinHashing (Python)

Locality sensitive hashing (LSH) is a widely popular technique used in approximate nearest neighbor (ANN) search. The solution to efficient similarity search is a profitable one - it is at the core of several billion (and even trillion) dollar companies.

LSH consists of a variety of different methods. In this video, we'll be covering the traditional approach - which consists of multiple steps - shingling, MinHashing, and the final banded LSH function.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing/

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff

00:00 Intro
01:21 Overview
05:58 Shingling
08:45 Vocab
09:27 One-hot Encoding
11:10 MinHash
15:51 Signature Info
18:08 LSH
22:20 Tuning LSH


How LSH Random Projection works in search (+Python)

Locality sensitive hashing (LSH) is a widely popular technique used in approximate similarity search. The solution to efficient similarity search is a profitable one - it is at the core of several billion (and even trillion) dollar companies.

The problem with similarity search is scale. Many companies deal with millions-to-billions of data points every single day. Given a billion data points, is it feasible to compare all of them with every search?

Further, many companies are not performing single searches - Google deals with more than 3.8 million searches every minute.

Billions of data points combined with high-frequency searches are problematic - and we haven't considered the dimensionality nor the similarity function itself. Clearly, an exhaustive search across all data points is unrealistic for larger datasets.

The solution to searching impossibly huge datasets? Approximate search. Rather than exhaustively comparing every pair, we approximate - restricting the search scope only to high probability matches.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing-random-projection/

Download Sift1M:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

IndexLSH for Fast Similarity Search in Faiss:
https://youtu.be/ZLfdQq_u7Eo

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff


IndexLSH for Fast Similarity Search in Faiss

Faiss  -  or Facebook AI Similarity Search  -  is an open-source framework built for enabling similarity search.

Faiss has many super-efficient implementations of different indexes that we can use in similarity search. That long list of indexes includes IndexLSH - an easy-to-use implementation of everything we have covered so far in LSH.

🌲 Pinecone article:
https://www.pinecone.io/learn/locality-sensitive-hashing-random-projection/

Download Sift1M:
https://gist.github.com/jamescalam/a09a16c17b677f2cf9c019114711f3bf

How LSH Random Projection works in search (+Python):
https://youtu.be/8bOrMqEdfiQ

🤖 70% Discount on the NLP With Transformers in Python course:
https://bit.ly/3DFvvY5

🎉 Sign-up For New Articles Every Week on Medium!
https://medium.com/@jamescalam/membership

👾 Discord:
https://discord.gg/c5QtDB9RAP

🕹 Free AI-Powered Code Refactoring with Sourcery:
https://sourcery.ai/?utm_source=YouTub&utm_campaign=JBriggs&utm_medium=aff