# Getting Started with Aggregation Pipelines in Rust

    

MongoDB's aggregation pipelines are one of its most powerful features. They allow you to write expressions, broken down into a series of stages, which perform operations including aggregation, transformations, and joins on the data in your MongoDB databases. This allows you to do calculations and analytics across documents and collections within your MongoDB database.

## Prerequisites

This quick start is the second in a series of Rust posts. I *highly* recommend you start with my first post, Basic MongoDB Operations in Rust, which will show you how to get set up correctly with a free MongoDB Atlas database cluster containing the sample data you'll be working with here. Go read it and come back. I'll wait. Without it, you won't have the database set up correctly to run the code in this quick start guide.

In summary, you'll need:

-   An up-to-date version of Rust. I used 1.49, but any recent version should work well.
-   A code editor of your choice. I recommend VS Code with the Rust Analyzer extension.
-   A MongoDB cluster containing the `sample_mflix` dataset. You can find instructions to set that up in the first blog post in this series.

## Getting Started

MongoDB's aggregation pipelines are very powerful and so they can seem a little overwhelming at first. For this reason, I'll start off slowly.  First, I'll show you how to build up a pipeline that duplicates behaviour that you can already achieve with MongoDB's `find()` method, but instead using an aggregation pipeline with `$match`, `$sort`, and `$limit` stages. Then, I'll show how to make queries that go beyond what can be done with `find`, demonstrating using `$lookup` to include related documents from another collection. Finally, I'll put the "aggregation" into "aggregation pipeline" by showing you how to use `$group` to group together documents to form new document summaries.

>All of the sample code for this quick start series can be found on GitHub. I recommend you check it out if you get stuck, but otherwise, it's worth following the tutorial and writing the code yourself!

All of the pipelines in this post will be executed against the sample_mflix database's `movies` collection. It contains documents that look like this (I'm showing you what they look like in Python, because it's a little more readable than the equivalent Rust struct):

``` python
{
   '_id': ObjectId('573a1392f29313caabcdb497'),
   'awards': {'nominations': 7,
               'text': 'Won 1 Oscar. Another 2 wins & 7 nominations.',
               'wins': 3},
   'cast': 'Janet Gaynor', 'Fredric March', 'Adolphe Menjou', 'May Robson'],
   'countries': ['USA'],
   'directors': ['William A. Wellman', 'Jack Conway'],
   'fullplot': 'Esther Blodgett is just another starry-eyed farm kid trying to '
               'break into the movies. Waitressing at a Hollywood party, she '
               'catches the eye of alcoholic star Norman Maine, is given a test, '
               'and is caught up in the Hollywood glamor machine (ruthlessly '
               'satirized). She and her idol Norman marry; but his career '
               'abruptly dwindles to nothing',
   'genres': ['Drama'],
   'imdb': {'id': 29606, 'rating': 7.7, 'votes': 5005},
   'languages': ['English'],
   'lastupdated': '2015-09-01 00:55:54.333000000',
   'plot': 'A young woman comes to Hollywood with dreams of stardom, but '
            'achieves them only with the help of an alcoholic leading man whose '
            'best days are behind him.',
   'poster': 'https://m.media-amazon.com/images/M/MV5BMmE5ODI0NzMtYjc5Yy00MzMzLTk5OTQtN2Q3MzgwOTllMTY3XkEyXkFqcGdeQXVyNjc0MzMzNjA@._V1_SY1000_SX677_AL_.jpg',
   'rated': 'NOT RATED',
   'released': datetime.datetime(1937, 4, 27, 0, 0),
   'runtime': 111,
   'title': 'A Star Is Born',
   'tomatoes': {'critic': {'meter': 100, 'numReviews': 11, 'rating': 7.4},
               'dvd': datetime.datetime(2004, 11, 16, 0, 0),
               'fresh': 11,
               'lastUpdated': datetime.datetime(2015, 8, 26, 18, 58, 34),
               'production': 'Image Entertainment Inc.',
               'rotten': 0,
               'viewer': {'meter': 79, 'numReviews': 2526, 'rating': 3.6},
               'website': 'http://www.vcientertainment.com/Film-Categories?product_id=73'},
   'type': 'movie',
   'writers': ['Dorothy Parker (screen play)',
               'Alan Campbell (screen play)',
               'Robert Carson (screen play)',
               'William A. Wellman (from a story by)',
               'Robert Carson (from a story by)'],
   'year': 1937}
```

There's a lot of data there, but I'll be focusing mainly on the `_id`, `title`, `year`, and `cast` fields.

## Your First Aggregation Pipeline

Aggregation pipelines are executed by the mongodb module using a Collection's [aggregate() method.

The first argument to `aggregate()` is a sequence of pipeline stages to be executed. Much like a query, each stage of an aggregation pipeline is a BSON document.  You'll often create these using the `doc!` macro that was introduced in the previous post.

An aggregation pipeline operates on *all* of the data in a collection.  Each stage in the pipeline is applied to the documents passing through, and whatever documents are emitted from one stage are passed as input to the next stage, until there are no more stages left. At this point, the documents emitted from the last stage in the pipeline are returned to the client program, as a cursor, in a similar way to a call to `find()`.

Individual stages, such as `$match`, can act as a filter, to only pass through documents matching certain criteria. Other stage types, such as `$project`, `$addFields`, and `$lookup`, will modify the content of individual documents as they pass through the pipeline. Finally, certain stage types, such as `$group`, will create an entirely new set of documents based on the documents passed into it taken as a whole. None of these stages change the data that is stored in MongoDB itself. They just change the data before returning it to your program! There *are* stages, like $out, which can save the results of a pipeline back into MongoDB, but I won't be covering it in this quick start.

I'm going to assume that you're working in the same environment that you used for the last post, so you should already have the mongodb crate configured as a dependency in your `Cargo.toml` file, and you should have a `.env` file containing your `MONGODB_URI` environment variable.

### Finding and Sorting

First, paste the following into your Rust code:

``` rust
// Load the MongoDB connection string from an environment variable:
let client_uri =
   env::var("MONGODB_URI").expect("You must set the MONGODB_URI environment var!");

// An extra line of code to work around a DNS issue on Windows:
let options =
   ClientOptions::parse_with_resolver_config(&client_uri, ResolverConfig::cloudflare())
      .await?;
let client = mongodb::Client::with_options(options)?;

// Get the 'movies' collection from the 'sample_mflix' database:
let movies = client.database("sample_mflix").collection("movies");
```

The above code will provide a `Collection` instance called `movie_collection`, which points to the `movies` collection in your database.

Here is some code which creates a pipeline, executes it with `aggregate`, and then loops through and prints the detail of each movie in the results. Paste it into your program.

``` rust
// Usually implemented outside your main function:
#derive(Deserialize)]
struct MovieSummary {
   title: String,
   cast: Vec,
   year: i32,
}

impl fmt::Display for MovieSummary {
   fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
      write!(
         f,
         "{}, {}, {}",
         self.title,
         self.cast.get(0).unwrap_or(&"- no cast -".to_owned()),
         self.year
      )
   }
}

// Inside main():
let pipeline = vec![
   doc! {
      // filter on movie title:
      "$match": {
         "title": "A Star Is Born"
      }
   },
   doc! {
      // sort by year, ascending:
      "$sort": {
         "year": 1
      }
   },
];

// Look up "A Star is Born" in ascending year order:
let mut results = movies.aggregate(pipeline, None).await?;
// Loop through the results, convert each of them to a MovieSummary, and then print out.
while let Some(result) = results.next().await {
   // Use serde to deserialize into the MovieSummary struct:
   let doc: MovieSummary = bson::from_document(result?)?;
   println!("* {}", doc);
}
```

This pipeline has two stages. The first is a [$match stage, which is similar to querying a collection with `find()`. It filters the documents passing through the stage based on a read operation query. Because it's the first stage in the pipeline, its input is all of the documents in the `movie` collection. The query for the `$match` stage filters on the `title` field of the input documents, so the only documents that will be output from this stage will have a title of "A Star Is Born."

The second stage is a $sort stage. Only the documents for the movie "A Star Is Born" are passed to this stage, so the result will be all of the movies called "A Star Is Born," now sorted by their year field, with the oldest movie first.

Calls to aggregate() return a cursor pointing to the resulting documents. Cursor implements the Stream trait.  The cursor can be looped through like any other stream, as long as you've imported StreamExt, which provides the `next()` method. The code above loops through all of the returned documents and prints a short summary, consisting of the title, the first actor in the `cast` array, and the year the movie was produced.

Executing the code above results in:

``` none
* A Star Is Born, Janet Gaynor, 1937
* A Star Is Born, Judy Garland, 1954
* A Star Is Born, Barbra Streisand, 1976
```

### Refactoring the Code

It is possible to build up whole aggregation pipelines as a single data structure, as in the example above, but it's not necessarily a good idea. Pipelines can get long and complex. For this reason, I recommend you build up each stage of your pipeline as a separate variable, and then combine the stages into a pipeline at the end, like this:

``` rust
// Match title = "A Star Is Born":
let stage_match_title = doc! {
   "$match": {
      "title": "A Star Is Born"
   }
};

// Sort by year, ascending:
let stage_sort_year_ascending = doc! {
   "$sort": { "year": 1 }
};

// Now the pipeline is easier to read:
let pipeline = vec stage.

The **modified and new** code looks like this:

``` rust
// Sort by year, descending:
let stage_sort_year_descending = doc! {
   "$sort": {
      "year": -1
   }
};

// Limit to 1 document:
let stage_limit_1 = doc! { "$limit": 1 };

let pipeline = vec stage.

I'll show you how to obtain related documents from another collection, and embed them in the documents from your primary collection.

First, modify the definition of the `MovieSummary` struct so that it has a `comments` field, loaded from a `related_comments` BSON field. Define a `Comment` struct that contains a subset of the data contained in a `comments` document.

``` rust
#derive(Deserialize)]
struct MovieSummary {
   title: String,
   cast: Vec,
   year: i32,
   #[serde(default, rename = "related_comments")]
   comments: Vec,
}

#[derive(Debug, Deserialize)]
struct Comment {
   email: String,
   name: String,
   text: String,
}
```

Next, create a new pipeline from scratch, and start with the following:

``` rust
// Look up related documents in the 'comments' collection:
let stage_lookup_comments = doc! {
   "$lookup": {
      "from": "comments",
      "localField": "_id",
      "foreignField": "movie_id",
      "as": "related_comments",
   }
};

// Limit to the first 5 documents:
let stage_limit_5 = doc! { "$limit": 5 };

let pipeline = vec![
   stage_lookup_comments,
   stage_limit_5,
];

let mut results = movies.aggregate(pipeline, None).await?;
// Loop through the results and print a summary and the comments:
while let Some(result) = results.next().await {
   let doc: MovieSummary = bson::from_document(result?)?;
   println!("* {}, comments={:?}", doc, doc.comments);
}
```

The stage I've called `stage_lookup_comments` is a `$lookup` stage. This `$lookup` stage will look up documents from the `comments` collection that have the same movie id. The matching comments will be listed as an array in a BSON field named `related_comments`, with an array value containing all of the comments that have this movie's `_id` value as `movie_id`.

I've added a `$limit` stage just to ensure that there's a reasonable amount of output without being overwhelming.

Now, execute the code.

>
>
>You may notice that the pipeline above runs pretty slowly! There are two reasons for this:
>
>-   There are 23.5k movie documents and 50k comments.
>-   There's a missing index on the `comments` collection. It's missing on purpose, to teach you about indexes!
>
>I'm not going to show you how to fix the index problem right now. I'll write about that in a later post in this series, focusing on indexes.  Instead, I'll show you a trick for working with slow aggregation pipelines while you're developing.
>
>Working with slow pipelines is a pain while you're writing and testing the pipeline. *But*, if you put a temporary `$limit` stage at the *start* of your pipeline, it will make the query faster (although the results may be different because you're not running on the whole dataset).
>
>When I was writing this pipeline, I had a first stage of `{ "$limit": 1000 }`.
>
>When you have finished crafting the pipeline, you can comment out the first stage so that the pipeline will now run on the whole collection.  **Don't forget to remove the first stage, or you're going to get the wrong results!**
>
>

The aggregation pipeline above will print out summaries of five movie documents. I expect that most or all of your movie summaries will end with this: `comments=[]`.

### Matching on Array Length

If you're *lucky*, you may have some documents in the array, but it's unlikely, as most of the movies have no comments. Now, I'll show you how to add some stages to match only movies which have more than two comments.

Ideally, you'd be able to add a single `$match` stage which obtained the length of the `related_comments` field and matched it against the expression `{ "$gt": 2 }`. In this case, it's actually two steps:

-   Add a field (I'll call it `comment_count`) containing the length of the `related_comments` field.
-   Match where the value of `comment_count` is greater than two.

Here is the code for the two stages:

``` rust
// Calculate the number of comments for each movie:
let stage_add_comment_count = doc! {
   "$addFields": {
      "comment_count": {
         "$size": "$related_comments"
      }
   }
};

// Match movie documents with more than 2 comments:
let stage_match_with_comments = doc! {
   "$match": {
      "comment_count": {
         "$gt": 2
      }
   }
};
```

The two stages go after the `$lookup` stage, and before the `$limit` 5 stage:

``` rust
let pipeline = vec![
   stage_lookup_comments,
   stage_add_comment_count,
   stage_match_with_comments,
   limit_5,
]
```

While I'm here, I'm going to clean up the output of this code to format the comments slightly better:

``` rust
let mut results = movies.aggregate(pipeline, None).await?;
// Loop through the results and print a summary and the comments:
while let Some(result) = results.next().await {
   let doc: MovieSummary = bson::from_document(result?)?;
   println!("* {}", doc);
   if doc.comments.len() > 0 {
      // Print a max of 5 comments per movie:
      for comment in doc.comments.iter().take(5) {
         println!(
            "  - {} <{}>: {}",
            comment.name,
            comment.email,
            comment.text.chars().take(60).collect::(),
         );
      }
   } else {
      println!("  - No comments");
   }
}
```

*Now* when you run this code, you should see something more like this:

``` none
* Midnight, Claudette Colbert, 1939
  - Sansa Stark : Error ex culpa dignissimos assumenda voluptates vel. Qui inventore 
  - Theon Greyjoy : Animi dolor minima culpa sequi voluptate. Possimus necessitatibu
  - Donna Smith : Et esse nulla ducimus tempore aliquid. Suscipit iste dignissimos v
```

It's good to see Sansa Stark from Game of Thrones really knows her Latin, isn't it?

Now I've shown you how to work with lookups in your pipelines, I'll show you how to use the `$group` stage to do actual *aggregation*.

## Grouping Documents with `$group`

I'll start with a new pipeline again.

The `$group` stage is one of the more difficult stages to understand, so I'll break this down slowly.

Start with the following code:

``` rust
// Define a struct to hold grouped data by year:
#[derive(Debug, Deserialize)]
struct YearSummary {
   _id: i32,
   #[serde(default)]
   movie_count: i64,
   #[serde(default)]
   movie_titles: Vec,
}

// Some movies have "year" values ending with 'è'.
// This stage will filter them out:
let stage_filter_valid_years = doc! {
   "$match": {
      "year": {
         "$type": "number",
      }
   }
};

/*
* Group movies by year, producing 'year-summary' documents that look like:
* {
*     '_id': 1917,
* }
*/
let stage_group_year = doc! {
   "$group": {
      "_id": "$year",
   }
};

let pipeline = vec![stage_filter_valid_years, stage_group_year];

// Loop through the 'year-summary' documents:
let mut results = movies.aggregate(pipeline, None).await?;
// Loop through the yearly summaries and print their debug representation:
while let Some(result) = results.next().await {
   let doc: YearSummary = bson::from_document(result?)?;
   println!("* {:?}", doc);
}
```

In the `movies` collection, some of the years contain the "è" character.  This database has some messy values in it. In this case, there's only a small handful of documents, and I think we should just remove them, so I've added a `$match` stage that filters out any documents with a `year` that's not numeric.

Execute this code, and you should see something like this:

``` none
* YearSummary { _id: 1959, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1980, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1977, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1933, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1998, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1922, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1948, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1965, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1950, movie_count: 0, movie_titles: [] }
* YearSummary { _id: 1968, movie_count: 0, movie_titles: [] }
...
```

Each line is a document emitted from the aggregation pipeline. But you're not looking at *movie* documents anymore. The `$group` stage groups input documents by the specified `_id` expression and outputs one document for each unique `_id` value. In this case, the expression is `$year`, which means one document will be emitted for each unique value of the `year` field. Each document emitted can (and usually will) also contain values generated from aggregating data from the grouped documents. Currently, the YearSummary documents are using the default values for `movie_count` and `movie_titles`. Let's fix that.

Change the stage definition to the following:

``` rust
let stage_group_year = doc! {
   "$group": {
      "_id": "$year",
      // Count the number of movies in the group:
      "movie_count": { "$sum": 1 },
   }
};
```

This will add a `movie_count` field, containing the result of adding `1` for every document in the group. In other words, it counts the number of movie documents in the group. If you execute the code now, you should see something like the following:

``` none
* YearSummary { _id: 2005, movie_count: 758, movie_titles: [] }
* YearSummary { _id: 1999, movie_count: 542, movie_titles: [] }
* YearSummary { _id: 1943, movie_count: 36, movie_titles: [] }
* YearSummary { _id: 1926, movie_count: 9, movie_titles: [] }
* YearSummary { _id: 1935, movie_count: 40, movie_titles: [] }
* YearSummary { _id: 1966, movie_count: 116, movie_titles: [] }
* YearSummary { _id: 1971, movie_count: 116, movie_titles: [] }
* YearSummary { _id: 1952, movie_count: 58, movie_titles: [] }
* YearSummary { _id: 2013, movie_count: 1221, movie_titles: [] }
* YearSummary { _id: 1912, movie_count: 2, movie_titles: [] }
...
```

There are a number of [accumulator operators, like `$sum`, that allow you to summarize data from the group. If you wanted to build an array of all the movie titles in the emitted document, you could add `"movie_titles": { "$push": "$title" },` to the `$group` stage. In that case, you would get `YearSummary` instances that look like this:

``` none
* YearSummary { _id: 1986, movie_count: 206, movie_titles: "Defense of the Realm", "F/X", "Mala Noche", "Witch from Nepal", ... ]}
```

Add the following stage to sort the results:

``` rust
let stage_sort_year_ascending = doc! {
   "$sort": {"_id": 1}
};

let pipeline = vec! [
   stage_filter_valid_years,  // Match numeric years
   stage_group_year,
   stage_sort_year_ascending, // Sort by year (which is the unique _id field)
]
```

Note that the `$match` stage is added to the start of the pipeline, and the `$sort` is added to the end. A general rule is that you should filter documents out early in your pipeline, so that later stages have fewer documents to deal with. It also ensures that the pipeline is more likely to be able to take advantages of any appropriate indexes assigned to the collection.

>Remember, all of the sample code for this quick start series can be found [on GitHub.

Aggregations using `$group` are a great way to discover interesting things about your data. In this example, I'm illustrating the number of movies made each year, but it would also be interesting to see information about movies for each country, or even look at the movies made by different actors.

## What Have You Learned?

You've learned how to construct aggregation pipelines to filter, group, and join documents with other collections. You've hopefully learned that putting a `$limit` stage at the start of your pipeline can be useful to speed up development (but should be removed before going to production).  You've also learned some basic optimization tips, like putting filtering expressions towards the start of your pipeline instead of towards the end.

As you've gone through, you'll probably have noticed that there's a *ton* of different stage types, operators, and accumulator operators.  Learning how to use the different components of aggregation pipelines is a big part of learning to use MongoDB effectively as a developer.

I love working with aggregation pipelines, and I'm always surprised at what you can do with them!

## Next Steps

Aggregation pipelines are super powerful, and because of this, they're a big topic to cover. Check out the full documentation to get a better idea of their full scope.

MongoDB University also offers a *free* online course on The MongoDB Aggregation Framework.

Note that aggregation pipelines can also be used to generate new data and write it back into a collection, with the $out stage.

MongoDB provides a *free* GUI tool called Compass. It allows you to connect to your MongoDB cluster, so you can browse through databases and analyze the structure and contents of your collections. It includes an aggregation pipeline builder which makes it easier to build aggregation pipelines. I highly recommend you install it, or if you're using MongoDB Atlas, use its similar aggregation pipeline builder in your browser. I often use them to build aggregation pipelines, and they include export buttons which will export your pipeline as Python code (which isn't too hard to transform into Rust).

I don't know about you, but when I was looking at some of the results above, I thought to myself, "It would be fun to visualise this with a chart." MongoDB provides a hosted service called Charts which just *happens* to take aggregation pipelines as input. So, now's a good time to give it a try!