# IoT and MongoDB: Powering Time Series Analysis of Household Power Consumption

IoT (Internet of Things) systems are increasingly becoming a part of our daily lives, offering smart solutions for homes and businesses. 

This article will explore a practical case study on household power consumption, showcasing how MongoDB's time series collections can be leveraged to store, manage, and analyze data generated by IoT devices efficiently.

## Time series collections

Time series collections in MongoDB effectively store time series data — a sequence of data points analyzed to observe changes over time.

Time series collections provide the following benefits:

- Reduced complexity for working with time series data
- Improved query efficiency
- Reduced disk usage
- Reduced I/O for read operations
- Increased WiredTiger cache usage

Generally, time series data is composed of the following elements:

- The timestamp of each data point
- Metadata (also known as the source), which is a label or tag that uniquely identifies a series and rarely changes
- Measurements (also known as metrics or values), representing the data points tracked at increments in time — generally key-value pairs that change over time

## Case study: household electric power consumption

This case study focuses on analyzing the data set with over two million data points of household electric power consumption, with a one-minute sampling rate over almost four years.

The dataset includes the following information:

- **date**: Date in format dd/mm/yyyy 
- **time**: Time in format hh:mm:ss 
- **global_active_power**: Household global minute-averaged active power (in kilowatt) 
- **global_reactive_power**: Household global minute-averaged reactive power (in kilowatt) 
- **voltage**: Minute-averaged voltage (in volt) 
- **global_intensity**: Household global minute-averaged current intensity (in ampere) 
- **sub_metering_1**: Energy sub-metering No. 1 (in watt-hour of active energy); corresponds to the kitchen, containing mainly a dishwasher, an oven, and a microwave (hot plates are not electric but gas-powered) 
- **sub_metering_2**: Energy sub-metering No. 2 (in watt-hour of active energy); corresponds to the laundry room, containing a washing machine, a tumble drier, a refrigerator, and a light. 
- **sub_metering_3**: Energy sub-metering No. 3 (in watt-hour of active energy); corresponds to an electric water heater and an air conditioner

## Schema modeling

To define and model our time series collection, we will use the Mongoose library. Mongoose, an Object Data Modeling (ODM) library for MongoDB, is widely used in the Node.js ecosystem for its ability to provide a straightforward way to model our application data.

The schema will include:

- **timestamp:** A combination of the “date” and “time” fields from the dataset.
- **global_active_power**: A numerical representation from the dataset.
- **global_reactive_power**: A numerical representation from the dataset. 
- **voltage**: A numerical representation from the dataset. 
- **global_intensity**: A numerical representation from the dataset.
- **sub_metering_1**: A numerical representation from the dataset. 
- **sub_metering_2**: A numerical representation from the dataset.
- **sub_metering_3**: A numerical representation from the dataset.

To configure the collection as a time series collection, an additional “**timeseries**” configuration with “**timeField**” and “**granularity**” properties is necessary. The “**timeField**” will use our schema’s “**timestamp**” property, and “**granularity**” will be set to “minutes” to match the dataset's sampling rate.

Additionally, an index on the “timestamp” field will be created to enhance query performance — note that you can query a time series collection the same way you query a standard MongoDB collection.

The resulting schema is structured as follows:

```javascript
const { Schema, model } = require('mongoose');

const powerConsumptionSchema = new Schema(
    {
        timestamp: { type: Date, index: true },
        global_active_power: { type: Number },
        global_reactive_power: { type: Number },
        voltage: { type: Number },
        global_intensity: { type: Number },
        sub_metering_1: { type: Number },
        sub_metering_2: { type: Number },
        sub_metering_3: { type: Number },
    },
    {
        timeseries: {
            timeField: 'timestamp',
            granularity: 'minutes',
        },
    }
);

const PowerConsumptions = model('PowerConsumptions', powerConsumptionSchema);

module.exports = PowerConsumptions;
```

For further details on creating time series collections, refer to MongoDB's official time series documentation.

## Inserting data to MongoDB

The dataset is provided as a .txt file, which is not directly usable with MongoDB. To import this data into our MongoDB database, we need to preprocess it so that it aligns with our database schema design.

This can be accomplished by performing the following steps:

1. Connect to MongoDB.
2. Load data from the .txt file.
3. Normalize the data and split the content into lines.
4. Parse the lines into structured objects.
5. Transform the data to match our MongoDB schema model.
6. Filter out invalid data.
7. Insert the final data into MongoDB in chunks.

Here is the Node.js script that automates these steps:

```javascript
// Load environment variables from .env file
require('dotenv').config();

// Import required modules
const fs = require('fs');
const mongoose = require('mongoose');
const PowerConsumptions = require('./models/power-consumption');

// Connect to MongoDB and process the data file
const processData = async () => {
    try {
        // Connect to MongoDB using the connection string from environment variables
        await mongoose.connect(process.env.MONGODB_CONNECTION_STRING);

        // Define the file path for the data source
        const filePath = 'Household_Power_Consumption.txt';

        // Read data file
        const rawFileContent = fs.readFileSync(filePath, 'utf8');

        // Normalize line endings and split the content into lines
        const lines = rawFileContent.replace(/\r\n/g, '\n').replace(/\r/g, '\n').trim().split('\n');

        // Extract column headers
        const headers = lines0].split(';').map((header) => header.trim());

        // Parse the lines into structured objects
        const parsedRecords = lines.slice(1).map((line) => {
            const values = line.split(';').map((value) => value.trim());
            return headers.reduce((object, header, index) => {
                object[header] = values[index];
                return object;
            }, {});
        });

        // Transform and prepare data for insertion
        const transformedRecords = parsedRecords.map((item) => {
            const [day, month, year] = item.Date.split('/').map((num) => parseInt(num, 10));
            const [hour, minute, second] = item.Time.split(':').map((num) => parseInt(num, 10));
            const dateObject = new Date(year, month - 1, day, hour, minute, second);

            return {
                timestamp: dateObject.toISOString(),
                global_active_power: parseFloat(item.Global_active_power),
                global_reactive_power: parseFloat(item.Global_reactive_power),
                voltage: parseFloat(item.Voltage),
                global_intensity: parseFloat(item.Global_intensity),
                sub_metering_1: parseFloat(item.Sub_metering_1),
                sub_metering_2: parseFloat(item.Sub_metering_2),
                sub_metering_3: parseFloat(item.Sub_metering_3),
            };
        });

        // Filter out invalid data
        const finalData = transformedRecords.filter(
            (item) =>
            item.timestamp !== 'Invalid Date' &&
            !isNaN(item.global_active_power) &&
            !isNaN(item.global_reactive_power) &&
            !isNaN(item.voltage) &&
            !isNaN(item.global_intensity) &&
            !isNaN(item.sub_metering_1) &&
            !isNaN(item.sub_metering_2) &&
            !isNaN(item.sub_metering_3)
        );

        // Insert final data into the database in chunks of 1000
        const chunkSize = 1000;
        for (let i = 0; i < finalData.length; i += chunkSize) {
            const chunk = finalData.slice(i, i + chunkSize);
            await PowerConsumptions.insertMany(chunk);
        }

        console.log('Data processing and insertion completed.');
    } catch (error) {
        console.error('An error occurred:', error);
    }
};

// Call the processData function
processData();
```

Before you start the script, you need to make sure that your environment variables are set up correctly. To do this, create a file named “.env” in the root folder, and add a line for “MONGODB_CONNECTION_STRING”, which is your link to the MongoDB database. 

The content of the .env file should look like this:

```javascript
MONGODB_CONNECTION_STRING = 'mongodb+srv://{{username}}:{{password}}@{{your_cluster_url}}/{{your_database}}?retryWrites=true&w=majority'
```

For more details on constructing your connection string, refer to the [official MongoDB documentation.

## Visualization with MongoDB Atlas Charts

Once the data has been inserted into our MongoDB time series collection, MongoDB Atlas Charts can be used to effortlessly connect to and visualize the data.

In order to connect and use MongoDB Atlas Charts, we should:

1. Establish a connection to the time series collection as a data source.
2. Associate the desired fields with the appropriate X and Y axes.
3. Implement filters as necessary to refine the data displayed.
4. Explore the visualizations provided by Atlas Charts to gain insights.

 to share your experiences, ask questions, and collaborate with fellow enthusiasts. Whether you are seeking advice, sharing your latest project, or exploring innovative uses of MongoDB, the community is a great place to continue the conversation.

  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt50c50df158186ba2/65f8b9fad467d26c5f0bbf14/image1.png