# Audio Find - Atlas Vector Search for Audio

## Introduction

As we venture deeper into the realm of digital audio, the frontiers of music discovery are expanding. The pursuit for a more personalized audio experience has led us to develop a state-of-the-art music catalog system. This system doesn't just archive music; it understands it. By utilizing advanced sound embeddings and leveraging the power of MongoDB Atlas Vector Search, we've crafted an innovative platform that recommends songs not by genre or artist, but by the intrinsic qualities of the music itself.

This article was done together with a co-writer, Ran Shir, music composer and founder of  Cues Assets , a production music group. We have researched and developed the following architecture to allow businesses to take advantage of their audio materials for searches.

### Demo video for the main flow
:youtube]{vid=RJRy0-kEbik}

## System architecture overview

At the heart of this music catalog is a Python service, intricately detailed in our Django-based views.py. This service is the workhorse for generating sound embeddings, using the Panns-inference model to analyze and distill the unique signatures of audio files uploaded by users. Here's how our sophisticated system operates:

**Audio file upload and storage:**

A user begins by uploading an MP3 file through the application's front end. This file is then securely transferred to Amazon S3, ensuring that the user's audio is stored safely in the cloud.

**Sound embedding generation:**
When an audio file lands in our cloud storage, our Django service jumps into action. It downloads the file from S3, using the Python requests library, into a temporary storage on the server to avoid any data loss during processing.

**Normalization and embedding processing:**

The downloaded audio file is then processed to extract its features. Using librosa, a Python library for audio analysis, the service loads the audio file and passes it to our Panns-inference model. The model, running on a GPU for accelerated computation, computes a raw 4096 members embedding vector which captures the essence of the audio.

**Embedding normalization:**

The raw embedding is then normalized to ensure consistent comparison scales when performing similarity searches. This normalization step is crucial for the efficacy of vector search, enabling a fair and accurate retrieval of similar songs.

**MongoDB Atlas Vector Search integration:**

The normalized embedding is then ready to be ingested by MongoDB Atlas. Here, it's indexed alongside the metadata of the audio file in the "embeddings" field. This indexing is what powers the vector search, allowing the application to perform a K-nearest neighbor (KNN) search to find and suggest the songs most similar to the one uploaded by the user.

**User interaction and feedback:**

Back on the front end, the application communicates with the user, providing status updates during the upload process and eventually serving the results of the similarity search, all in a user-friendly and interactive manner.

![Sound Catalog Similarity Architecture

This architecture encapsulates a blend of cloud technology, machine learning, and database management to deliver a unique music discovery experience that's as intuitive as it is revolutionary.

## Uploading and storing MP3 files

The journey of an MP3 file through our system begins the moment a user selects a track for upload. The frontend of the application, built with user interaction in mind, takes the first file from the dropped files and prepares it for upload. This process is initiated with an asynchronous call to an endpoint that generates a signed URL from AWS S3. This signed URL is a token of sorts, granting temporary permission to upload the file directly to our S3 bucket without compromising security or exposing sensitive credentials.

### Frontend code for file upload

The frontend code, typically written in JavaScript for a web application, makes use of the `axios` library to handle HTTP requests. When the user selects a file, the code sends a request to our back end to retrieve a signed URL. With this URL, the file can be uploaded to S3. The application handles the upload status, providing real-time feedback to the user, such as "Uploading..." and then "Searching based on audio..." upon successful upload. This interactive feedback loop is crucial for user satisfaction and engagement.

```javascript
async uploadFiles(files) {
  const file = files0]; // Get the first file from the dropped files
  if (file) {
    try {
      this.imageStatus = "Uploading...";
      // Post a request to the backend to get a signed URL for uploading the file
      const response = await axios.post('https://[backend-endpoint]/getSignedURL', {
        fileName: file.name,
        fileType: file.type
      });
      const { url } = response.data;
      // Upload the file to the signed URL
      const resUpload = await axios.put(url, file, {
        headers: {
          'Content-Type': file.type
        }
      });
      console.log('File uploaded successfully');
      console.log(resUpload.data);

      this.imageStatus = "Searching based on image...";
      // Post a request to trigger the audio description generation
      const describeResponse = await axios.post('https://[backend-endpoint]/labelsToDescribe', {
        fileName: file.name
      });

      const prompt = describeResponse.data;
      this.searchQuery = prompt;
      this.$refs.dropArea.classList.remove('drag-over');
      if (prompt === "I'm sorry, I can't provide assistance with that request.") {
        this.imageStatus = "I'm sorry, I can't provide assistance with that request."
        throw new Error("I'm sorry, I can't provide assistance with that request.");
      }
      this.fetchListings();
      // If the request is successful, show a success message
      this.showSuccessPopup = true;
      this.imageStatus = "Drag and drop an image here"

      // Auto-hide the success message after 3 seconds
      setTimeout(() => {
        this.showSuccessPopup = false;
      }, 3000);
    } catch (error) {
      console.error('File upload failed:', error);
      // In case of an error, reset the UI and show an error message
      this.$refs.dropArea.classList.remove('drag-over');
      this.showErrorPopup = true;

      // Auto-hide the error message after 3 seconds
      setTimeout(() => {
        this.showErrorPopup = false;
      }, 3000);

      // Reset the status message after 6 seconds
      setTimeout(() => {
        this.imageStatus = "Drag and drop an image here"
      }, 6000);

    }
  }
}
```

### Backend Code for Generating Signed URLs

On the backend, a Serverless function written for the MongoDB Realm platform interacts with AWS SDK. It uses stored AWS credentials to access S3 and create a signed URL, which it then sends back to the frontend. This URL contains all the necessary information for the file upload, including the file name, content type, and access control settings.

```javascript
// Serverless function to generate a signed URL for file uploads to AWS S3
exports = async function({ query, headers, body}, response) {
  
    // Import the AWS SDK
    const AWS = require('aws-sdk');

    // Update the AWS configuration with your access keys and region
    AWS.config.update({
      accessKeyId: context.values.get('YOUR_AWS_ACCESS_KEY'), // Replace with your actual AWS access key
      secretAccessKey: context.values.get('YOUR_AWS_SECRET_KEY'), // Replace with your actual AWS secret key
      region: 'eu-central-1' // The AWS region where your S3 bucket is hosted
    });
    
    // Create a new instance of the S3 service
    const s3 = new AWS.S3();
    // Parse the file name and file type from the request body
    const { fileName, fileType } = JSON.parse(body.text())
    
    // Define the parameters for the signed URL
    const params = {
    Bucket: 'YOUR_S3_BUCKET_NAME', // Replace with your actual S3 bucket name
    Key: fileName, // The name of the file to be uploaded
    ContentType: fileType, // The content type of the file to be uploaded
    ACL: 'public-read' // Access control list setting to allow public read access
  };
  
  // Generate the signed URL for the 'putObject' operation
  const url = await s3.getSignedUrl('putObject', params);
  
  // Return the signed URL in the response
  return { 'url' : url }
};
```

## Sound embedding with Panns-inference model
Once an MP3 file is securely uploaded to S3, a Python service, which interfaces with our Django back end, takes over. This service is where the audio file is transformed into something more — a compact representation of its sonic characteristics known as a sound embedding. Using the librosa library, the service reads the audio file, standardizing the sample rate to ensure consistency across all files. The Panns-inference model then takes a slice of the audio waveform and infers its embedding.

```python
import tempfile
from django.http import JsonResponse
from django.views.decorators.csrf import csrf_exempt
from panns_inference import AudioTagging
import librosa
import numpy as np
import os
import json
import requests

# Function to normalize a vector
def normalize(v):
    norm = np.linalg.norm(v)
    return v / norm if norm != 0 else v

# Function to generate sound embeddings from an audio file
def get_embedding(audio_file):
    # Initialize the AudioTagging model with the specified device
    model = AudioTagging(checkpoint_path=None, device='gpu')
    # Load the audio file with librosa, normalizing the sample rate to 44100
    a, _ = librosa.load(audio_file, sr=44100)
    # Add an extra dimension to the array to fit the model's input requirements
    query_audio = a[None, :]
    # Perform inference to get the embedding
    _, emb = model.inference(query_audio)
    # Normalize the embedding before returning
    return normalize(emb[0])

# Django view to handle the POST request for downloading and embedding
@csrf_exempt
def download_and_embed(request):
    if request.method == 'POST':
        try:
            # Parse the request body to get the file name
            body_data = json.loads(request.body.decode('utf-8'))
            file_name = body_data.get('file_name')

            # If the file name is not provided, return an error
            if not file_name:
                return JsonResponse({'error': 'Missing file_name in the request body'}, status=400)

            # Construct the file URL (placeholder) and send a request to get the file
            file_url = f"https://[s3-bucket-url].amazonaws.com/{file_name}"
            response = requests.get(file_url)

            # If the file is successfully retrieved
            if response.status_code == 200:
                # Create a temporary file to store the downloaded content
                with tempfile.NamedTemporaryFile(delete=False, suffix=".mp3") as temp_audio_file:
                    temp_audio_file.write(response.content)
                    temp_audio_file.flush()
                    # Log the temporary file's name and size for debugging
                    print(f"Temp file: {temp_audio_file.name}, size: {os.path.getsize(temp_audio_file.name)}")

                # Generate the embedding for the downloaded file
                embedding = get_embedding(temp_audio_file.name)
                # Return the embedding as a JSON response
                return JsonResponse({'embedding': embedding.tolist()})
            else:
                # If the file could not be downloaded, return an error
                return JsonResponse({'error': 'Failed to download the file'}, status=400)
        except json.JSONDecodeError:
            # If there is an error in the JSON data, return an error
            return JsonResponse({'error': 'Invalid JSON data in the request body'}, status=400)

    # If the request method is not POST, return an error
    return JsonResponse({'error': 'Invalid request'}, status=400)
```

### Role of Panns-inference model

The  [Panns-inference  model is a deep learning model trained to understand and capture the nuances of audio content. It generates a vector for each audio file, which is a numerical representation of the file's most defining features. This process turns a complex audio file into a simplified, quantifiable form that can be easily compared against others.

For more information and setting up this model see the following github example.

## Vector search with MongoDB Atlas

**Storing and indexing embeddings in MongoDB Atlas**

MongoDB Atlas is where the magic of searchability comes to life. The embeddings generated by our Python service are stored in a MongoDB Atlas collection. Atlas, with its robust indexing capabilities, allows us to index these embeddings efficiently, enabling rapid and accurate vector searches.
This is the index definition used on the “songs” collection:

```json
{
  "mappings": {
    "dynamic": false,
    "fields": {
      "embeddings": {
        "dimensions": 4096,
        "similarity": "dotProduct",
        "type": "knnVector"
      },
      "file": {
        "normalizer": "none",
        "type": "token"
      }
    }
  }
}
```
The "file" field is indexed with a "token" type for file name filtering logic, explained later in the article.

**Songs collection sample document:**
```json
{ 
_id : ObjectId("6534dd09164a19b0ac1f7311"),
 file : "Glorious Outcame Full Mix.mp3",
embeddings : Array (4096)]
}
```

### Vector search functionality

Vector search in MongoDB Atlas employs a K-nearest neighbor (KNN) algorithm to find the closest embeddings to the one provided by the user's uploaded file. When a user initiates a search, the system queries the Atlas collection, searching through the indexed embeddings to find and return a list of songs with the most similar sound profiles.
This combination of technologies — from the AWS S3 storage and signed URL generation to the processing power of the Panns-inference model, all the way to the search capabilities of MongoDB Atlas — creates a seamless experience. Users can not only upload their favorite tracks but also discover new ones that carry a similar auditory essence, all within an architecture built for scale, speed, and accuracy.

### Song Lookup and similarity search

**'“Get Songs” functionality**
The “Get Songs” feature is the cornerstone of the music catalog, enabling users to find songs with a similar auditory profile to their chosen track. When a user uploads a song, the system doesn't just store the file; it actively searches for and suggests tracks with similar sound embeddings. This is achieved through a similarity search, which uses the sound embeddings stored in the MongoDB Atlas collection.

```javascript
// Serverless function to perform a similarity search on the 'songs' collection in MongoDB Atlas
exports = async function({ query, body }, response) {
    // Initialize the connection to MongoDB Atlas
    const mongodb = context.services.get('mongodb-atlas');
    // Connect to the specific database
    const db = mongodb.db('YourDatabaseName'); // Replace with your actual database name
    // Connect to the specific collection within the database
    const songsCollection = db.collection('YourSongsCollectionName'); // Replace with your actual collection name

    // Parse the incoming request body to extract the embedding vector
    const parsedBody = JSON.parse(body.text());
    console.log(JSON.stringify(parsedBody)); // Log the parsed body for debugging

    // Perform a vector search using the parsed embedding vector
         let foundSongs = await songs.aggregate([
     { "$vectorSearch": {
       "index" : "default",
       "queryVector": parsedBody.embedding,
       "path": "embeddings",
       "numCandidates": 15,
       "limit" : 15
   }
  }
 ]).toArray()
   
    // Map the found songs to a more readable format by stripping unnecessary path components
    let searchableSongs = foundSongs.map((song) => {
        // Extract a cleaner, more readable song title
        let shortName = song.name.replace('.mp3', '');
        return shortName.replace('.wav', ''); // Handle both .mp3 and .wav file extensions
    });

    // Prepare an array of $unionWith stages to combine results from multiple collections if needed
    let unionWithStages = searchableSongs.slice(1).map((songTitle) => {
        return {
            $unionWith: {
                coll: 'RelatedSongsCollection', // Name of the other collection to union with
                pipeline: [
                    { $match: { "songTitleField": songTitle } }, // Match the song titles against the related collection
                ],
            },
        };
    });

    // Execute the aggregation query with a $match stage for the first song, followed by any $unionWith stages
    const relatedSongsCollection = db.collection('YourRelatedSongsCollectionName'); // Replace with your actual related collection name
    const locatedSongs = await relatedSongsCollection.aggregate([
        { $match: { "songTitleField": searchableSongs[0] } }, // Start with the first song's match stage
        ...unionWithStages, // Include additional stages for related songs
    ]).toArray();

    // Return the array of located songs as the response
    return locatedSongs;
};
```
Since embeddings are stored together with the songs data we can use the embedding field when performing a lookup of nearest N neighbours. This approach implements the "More Like This" button.

```javascript
// Get input song 3 neighbours which are not itself. "More Like This"
  let foundSongs = await songs.aggregate([
   { "$vectorSearch": {
    "index" : "default",
       "queryVector": songDetails.embeddings,
       "path": "embeddings",
       "filter" : { "file" : { "$ne" : fullSongName}},
       "numCandidates": 15,
       "limit" : 3
  }}
 ]).toArray()
``` 
The code [filter out the searched song itself.

## Backend code for similarity search

The backend code responsible for the similarity search is a serverless function within MongoDB Atlas. It executes an aggregation pipeline that begins with a vector search stage, leveraging the `$vectorSearch` operator with `queryVector` to perform a K-nearest neighbor search. The search is conducted on the "embeddings" field, comparing the uploaded track's embedding with those in the collection to find the closest matches. The results are then mapped to a more human-readable format, omitting unnecessary file path information for the user's convenience.
```javascript
    let foundSongs = await songs.aggregate(
     { "$vectorSearch": {
       "index" : "default",
       "queryVector": parsedBody.embedding,
       "path": "embeddings",
       "numCandidates": 15,
       "limit" : 15
   }
  }
 ]).toArray()
```

## Frontend functionality

**Uploading and searching for similar songs**

The front end provides a drag-and-drop interface for users to upload their MP3 files easily. Once a file is selected and uploaded, the front end communicates with the back end to initiate the search for similar songs based on the generated embedding. This process is made transparent to the user through real-time status updates.

** User Interface and Feedback Mechanisms **

The user interface is designed to be intuitive, with clear indications of the current process — whether it's uploading, searching, or displaying results. Success and error popups inform the user of the status of their request. A success popup confirms the upload and successful search, while an error popup alerts the user to any issues that occurred during the process. These popups are designed to auto-dismiss after a short duration to keep the interface clean and user-friendly.

## Challenges and solutions

### Developmental challenges

One of the challenges faced was ensuring the seamless integration of various services, such as AWS S3, MongoDB Atlas, and the Python service for sound embeddings. Handling large audio files and processing them efficiently required careful consideration of file management and server resources.

### Overcoming the challenges

To overcome these issues, we utilized temporary storage for processing and optimized the Python service to handle large files without significant memory overhead. Additionally, the use of serverless functions within MongoDB Atlas allowed us to manage compute resources effectively, scaling with the demand as needed.

## Conclusion

This music catalog represents a fusion of cloud storage, advanced audio processing, and modern database search capabilities. It offers an innovative way to explore music by sound rather than metadata, providing users with a uniquely tailored experience.

Looking ahead, potential improvements could include enhancing the [Panns-inference model for even more accurate embedding generation and expanding the database to accommodate a greater variety of audio content. Further refinements to the user interface could also be made, such as incorporating user feedback to improve the recommendation algorithm continually.
Looking ahead, potential improvements could include enhancing the model for even more accurate embedding generation and expanding the database to accommodate a greater variety of audio content. Further refinements to the user interface could also be made, such as incorporating user feedback to improve the recommendation algorithm continually.
In conclusion, the system stands as a testament to the possibilities of modern audio technology and database management, offering users a powerful tool for music discovery and promising avenues for future development.

**Special Thanks:** Ran Shir and Cues Assets group for the work, research efforts and materials.

Want to continue the conversation? Meet us over in the MongoDB Community forums!