Data Aggregation with MongoDB. MongoDB aggregation framework is a very…
By: Dummies Tech Blogger
Tags: NoSQL, Mongo, Aggregation, Mongodb Tutorial, Mongodb

MongoDB aggregation framework is a very powerful tool to do aggregation and analytics.

The aggregate queries are a series of command in pipeline that is given to MongoDB.

MongoDB goes on executing these queries one after another and each query in the pipeline is applied on the output result of the previous queries.

Here we look at the some of the most important and commonly used commands along with their usage and examples.

$project:

project helps to bring in a sub-set of fields that are required for analysis.

Suppose there are four fields in the document invoiceDate, invoiceQuantity, priceUnitPo and supplierPartNumber

You want to select only supplierPartNumber and priceUnitPo.

You can do:

{$project:{supplierPartNumber:1, priceUnitPo:1}}



Another way will be:

{$project:{spn: ‘$supplierPartNumber’, up: ‘$priceUnitPo’, dt: ‘$invoiceDate’}}//(<- up is short-form of Unit Price)

This will select the fields as “spn” and “up”. This will also reduce the data volume by shortening the field names.

To exclude these two fields and include any other fields you can do:

{$project:{supplierPartNumber:0, priceUnitPo:0}},



Or you can also do some calculation like selecting supplierPartNumber, invoiceDate in form of yyyy-mm-dd and totalOV by multiplying priceUnitPo with invoiceQuantity

{$project:{

spn: ‘$supplierPartNumber’, date: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //(format date in form of YYYY-mm-dd)

totalOv: {$multiply:[‘$priceUnitPo’, ‘$invoiceQuantity’]} //(calculate totalOV)

},

},

So, basically in SQL term $project is like creating a view out of the huge data for further analysis.

$match

match is like filtering data in the pipeline, based on a particular condition.

So, let us find all the supplierPartNumber which were ordered in a particular year (say 2020)

So, first I will project to get the supplierPartNumber and year. Then we match with the condition (year = 2020)

{$project:{

spn: ‘$supplierPartNumber’,

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //(format/extract date in form of YYYY)

totalOv: {$multiply:[‘$priceUnitPo’, ‘$invoiceQuantity’]} //(calculate totalOV)

},

}, //(->Project)

{

$match:{dateyy: ‘2020’} //(-> Match — date is now String)

},

$group:

group is used when data is aggregated against a certain field or against fields.

Here, let us find the total order values by year.

{$project:{

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //format date in form of YYYY

ov: {$multiply:[‘$priceUnitPo’, ‘$invoiceQuantity’]} //calculate totalOV

},

}, //(->Project. At the end of this we will have all the rows with dateyy as only the yyyy part of the date.)

{

$group:

{

_id: ‘$dateyy’, //Group by year

ov: {$sum: ‘$ov’} // Aggregation field and type of aggregation.

}

},

We can also have aggregation based on max, min.

But Mongo don’t have any count or count distinct. Let us see how we can achieve the.



First let us try to find the count of all the orders grouped by year (not sum of all order values).

{$project:{

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //format date in form of YYYY

},

}, //->Project At the end of this we will have all the rows with dateyy as only the yyyy part.

{

$group:

{

_id: ‘$dateyy’, //Group by year

qty: {$sum: 1} // ← Notice the change here. We sum up with 1.

}

},

And let us order it by number of orders descending

{$project:{

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //format date in form of YYYY

},

}, //->Project At the end of this we will have all the rows with dateyy as only the yyyy part.

{

$group:

{

_id: ‘$dateyy’, //Group by year

qty: {$sum: 1} // ← Notice the change here. We sum up with 1.

}

},

{

$sort : {qty:-1}

}

Now let us try to find the COUNT of DISTINCT number of supplierPartNumber yearwise.

For distinct count we need to first count by grouping the supplierPartNumber and year. This will give us count of each distinct combination of supplierPartNumber and year.

Then we discard this count and take only the list of distinct year and supplierPartNumber and aggregate the same by counting yearwise.

{$project:{

spn: ‘$supplierPartNumber’,

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //format date in form of YYYY

},

}, //->Project

{

$group:

{

_id: {dateyy: ‘$dateyy’, spn:’$spn’}, //<-Group by year and spn. This will give us distinct year and supplierPartNumber. This is all we need

qty: {$sum: 1} //<- We donot need this. We will discard this

},

},

{

$project:{

_id:0, //<-We donot need this

dateyy: ‘$_id.dateyy’, spn:‘$_id.spn’ //<- we require only the year and supplierPartNumber. So this peojection will give us the distinct combination of year and supplierPartNumber

},

},

{

$group:{

_id: ‘$dateyy’,//group by year

qty:{$sum: 1}//<- Since we already had distinct supplierPartNumber year-wise, grouping it against year will give distinct supplierPartNumber year-wise.

}

}

So COUNT DISTINCT in Mongo us basically two sets of aggregation as shown above.

One more thing. To find total order value (aggregated against nothing) we pass the _id as null.

{$project:{

dateyy: {$dateToString: { format: ‘%Y’, date: ‘$invoiceDate’ }}, //format date in form of YYYY

ov: {$multiply:[‘$priceUnitPo’, ‘$invoiceQuantity’]} //calculate totalOV

},

}, //->Project At the end of this we will have all the rows with dateyy as only the yyyy part.

{

$group:

{

_id: null, //Group by nothing!!!

ov: {$sum: ‘$ov’} // Aggregation field and type of aggregation.

}

},

Conclusion:

MongoDB offers a great set of facilities for aggregating data which can be further used in various reporting and analytics. Coupled with the advantages it has over traditional RDBMs, namely unstructured NoSQL, capability to handle hu”Mongo”us data and speed, MongoDB is truly a great tool for your reporting analytical needs.