MongoDB first sight guide
By: Pankaj Kushwaha
Tags: Database, Rdbms, Mongodb, NoSQL, Query Optimization

What is MongoDB?

First of all, the introduction of relational and non-relational databases;

for relational databases, before storing data, you must first build tables to build a database. As the complexity of the data increases, the number of tables created increases Many

but non-relational databases do not need to be so complicated;

Relational database is very powerful, but it cannot cope with all application scenarios well;

MySQL has poor scalability; IO pressure is high under big data; table structure changes are difficult.

MongoDB is a Non-Relational Data Store for JSON document. By Non-Relational we mean that it does not store data in tables like Relational Databases does. That is, MongoDB is a document database, in which each record in MongoDB is a document. The following is a summary of some important knowledge points of MongoDB database.

JSON: JavaScript Object Notation.

A very simple JSON document Looks like

{“FirstName”: “Pankaj”}

FirstName is the Key and the Value is Pankaj

MongoDB is Schema less

Schema less means two documents doesn’t need to have the same schema but we can create Dynamic Schema that can resolve on the fly.

You can create two similar document in the same collection as below.

{“FirstName”: “Pankaj”}

{“ FirstName”: “Pankaj”,“LastName”: “kumar”,“Location”: “Gurgaon”}

We know that MySQL is a relational database, but MongoDB is not. We use the following table for comparison:

Advantages and disadvantages of mongodb compared with relational databases

Weak consistency (eventually consistent), which can better guarantee the user’s access speed: For

Example, in a traditional relational database, a COUNT type operation will lock the data set, which can guarantee the comparison of the “current” situation. The exact value. This is very important in some cases, such as when checking account information through ATM, but for Wordnik, the data is constantly updated and growing. This kind of “precision” guarantee is almost meaningless, but it will produce a lot. Delay. What they need is an “approximate” number and faster processing speed.

But in some cases, MongoDB will lock the database. If there are hundreds of requests at this time, they will pile up and cause many problems. We use the following optimization method to avoid locking:

before each update, we will query the record first. The query operation will put the object into memory, so the update will be as fast as possible. In the master/slave deployment scheme, the slave node can be run with the “-pretouch” parameter, which can also get the same effect.

Use multiple mongod processes. We split the database into multiple processes based on access patterns.

②The storage method of the document structure can obtain data more conveniently.

For a hierarchical data structure, if such data is to be stored in a flat, tabular structure, it is very difficult to query or obtain data.

③Built-in GridFS to support large-capacity storage.

GridFS is an excellent distributed file system that can support massive data storage.

Built-in GridFS and MongoDB, can meet the fast range query of large data sets.

④ Built-in Sharding.

Provides a Range-based Auto Sharding mechanism: A collection can be divided into several segments according to the range of records, and divided into different shards.

Shards can be combined with replication, and with Replica sets, Sharding+fail-over can be achieved, and different Shards can be load balanced. The query is transparent to the client. The client performs operations such as query, statistics, and MapReduce, which will be automatically routed to the back-end data nodes by MongoDB. This allows us to focus on our business, and to upgrade painlessly when appropriate. The Sharding design capability of MongoDB is large enough to support about 20 petabytes, which is enough to support general applications.

This ensures that MongoDB runs on a cheap PC server cluster. The PC cluster is very convenient to expand and the cost is very low, avoiding the complexity and cost of “sharding” operation.

⑤ Rich third-party support. (This is an advantage MongoDB also has compared with other NoSQL)

Many NoSQL open source databases on the Internet are completely community-based, without official support, which brings great risks to users.

The open source document database MongoDB is backed by the commercial company 10gen to provide commercial training and support.

Moreover, the MongoDB community is very active, and many development frameworks quickly provided support for MongDB. Many well-known large companies and websites also use MongoDB in the production environment, and more and more innovative companies are turning to MongoDB as a technical solution with Django and RoR.

⑥ Superior performance:

In the case of use, with tens of millions of document objects and nearly 10G of data, queries on indexed IDs will not be slower than mysql, while queries on non-indexed fields will win overall. MySQL is actually not capable of querying any field under a large amount of data, but the query performance of mongodb really surprised me. The writing performance is also very satisfactory. For the same writing of millions of data, mongodb is much faster than the couchdb I have tried before, and it can be solved in less than 10 minutes. Add a sentence, mongodb is far from being a CPU killer during the observation process.

Compared with relational databases, MongoDB has the following disadvantages:

① Mongodb does not support transaction operations.

So a system with strict transaction requirements (if the banking system) definitely cannot use it. (This point corresponds to the advantage ①)

②Mongodb takes up too much space.

Regarding the reasons, the following aspects are mentioned in the official FAQ:

1. Space pre-allocation: In order to avoid excessive hard disk fragmentation, mongodb will apply to generate a large block of hard disk space every time the space is insufficient. Moreover, the amount of applications increased from 64M, 128M, 256M, until 2G is the larger volume of a single file. As the amount of data increases, you can see these files in the data directory with increasing capacity.

2. The space occupied by the field name: In order to keep the structure information in each record for query, mongodb needs to store the key-value of each field in the form of BSON. If the value field is not large relative to the key field, for example For storing numerical data, the overhead of the data is larger. One way to reduce the space occupation is to make the field name as short as possible, so that the space occupied is small, but this requires a trade-off between legibility and space occupation. I once suggested that the author make the field name an index, and each field name is represented by one byte, so that there is no need to worry about the length of the field name. But the author’s concern is not unreasonable. This indexing method needs to replace the index value with the original value after each query results, and then send it to the client. This replacement is also very time-consuming. The current realization is a trade of space for time.

3. Deleting records does not free up space: This is easy to understand. In order to avoid large-scale movement of data after the record is deleted, the original record space is not deleted, just mark “deleted”, and it can be reused later.

4. You can run db.repairDatabase() regularly to organize records, but this process will be slow

③MongoDB does not have a mature maintenance tool like MySQL, which is noteworthy for development and IT operations.



MongoDB is suitable for storing some data with simple relationships and a large amount of data, such as the monitoring information of virtual machines on our platform, including data such as memory, IO, CPU, and network. Data is collected every few seconds, weekly, every Month, the amount is very large, and the old monitoring data will not be retained for too long, use mongodb to store these data; in

addition, the cluster deployment of mongodb is relatively simple and easy to expand; such as master-slave replication, configure it in mongo.conf A few parameters are OK; the configuration of the sharded cluster is relatively simple. It also supports the use of the command line to dynamically add and delete nodes;



The advantages and disadvantages of Mongodb (1) the shortcomings of Mongodb

1. The data distribution in cluster





shards is uneven 2. The reliability of a single machine is relatively poor 3. Big data The amount of continuous insertion, the write performance fluctuates greatly 4. The disk space usage is relatively large

(2) Mongodb’s superiority

1. No mode

2. The query and indexing methods are flexible, and it is the Nosql that is most like SQL

3. Supports replication sets





Comparison of Mongodb and redis with features such as active standby, mutual active standby, automatic sharding, etc .:

mongoDB source language is C++, redis is also C or C++,

The mongodb file is stored in a BSON format similar to JSON, or a custom binary format.

The performance of mongodb and redis depends on the size of memory. Mongodb has rich data expression and index; most similar to relational database, supports rich query language, redis data is rich, less IO, mongodb has obvious advantages in this respect.

Mongodb does not support things. It depends on the client’s own guarantee. Redis supports things and is relatively weak. It can only ensure that the operations in the things are executed in order. In this respect, redis is better than mongodb.

Mongodb has improved the access efficiency of massive data, and redis’s performance and operation of smaller data volume are better than redis in this respect. Mongodb has mapredurce function and provides data analysis. Redis does not. In this respect, mongodb is better than redis.

Why use MongoDB?

(1) MongoDB proposes the concept of documents and collections. It uses BSON (JSON-like) as its data model structure. Its structure is object-oriented rather than a two-dimensional table. It is like this in MongoDB to store a user.

｛username:’123',password:’123'｝

Using this data model enables MongoDB to provide high read and write capabilities in a production environment, and the throughput is greatly enhanced compared to SQL databases such as mysql.

(2) Easy to scale, automatic failover. Easy scalability refers to the provision of sharding capabilities, the ability to shard a data set, and the data storage pressure to be shared among multiple servers. Automatic failover is the concept of a replica set. MongoDB can detect whether the master node is alive, and when inactive, it can automatically promote the slave node to the master node to achieve failover.

(3) Because the data model is object-oriented, it can express rich and hierarchical data structures. For example, in the blog system, you can directly add “comments” to the “article” document, instead of creating three sheets like myqsl Table to describe this relationship.

Main features

(1) The SQL database of the document data type is normalized, and the integrity and uniqueness of the data can be guaranteed through the constraints of the primary key or foreign key. Therefore, the SQL database is often used in systems with high data integrity. MongoDB is inferior to SQL-type databases in this respect, and MongoDB does not have a fixed Schema. It is precisely because MongoDB has fewer such constraints, which can make the data storage structure more flexible and the storage speed faster.

(2) Instant query ability

MongoDB retains the ability to query relational databases in real time, and retains the ability to index (the bottom layer is based on B tree). This draws on the advantages of relational databases. Compared with the same type of NoSQL redis, it does not have the above capabilities.

(3) Replication capability

MongoDB itself provides a replica set that can distribute data on multiple machines to achieve redundancy, the purpose is to provide automatic failover and expand read capabilities.

(4) Speed ​​and persistence The

MongoDB driver implements a write semantic fire and forget, that is, when the driver calls to write, you can immediately get a successful result (even if it is an error), which makes the writing speed faster. Of course, there will be a certain degree of insecurity, which is completely dependent on the network.

MongoDB provides the concept of Journaling log. In fact, it is like mysql’s bin-log log. When it needs to be inserted, it will write records into the log first, and then complete the actual data operation. In this way, if there is a power outage, the process is suddenly interrupted. You can ensure that the data will not be wrong, and you can read the Journaling log to repair it through the repair function.

(5) Data expansion

MongoDB uses fragmentation technology to expand data. MongoDB can automatically fragment and transfer data blocks in fragments, so that the data stored in each server is of the same size

Complete command line tools

mongodump and mongorestore, standard tools for backing up and restoring databases. Export the BSON format and migrate the database.

mongoexport and mongoimport are used to import and export JSON, CSV and TSV data, useful when the data needs to support multiple formats. mongoimport can also be used for the initial import of large data sets, but by the way, you should also pay attention to it before importing. In order to make full use of mongoDB, you usually need to make some adjustments to the data model.

mongosniff, a network sniffing tool, used to observe operations sent to the database. Basically, the BSON transmitted on the network is converted into a shell statement that is easy for people to read.

Therefore, it can be concluded that MongoDB combines the best features of key-value storage and relational databases. Because it is simple, the data is extremely fast, and it is relatively easy to scale a database that also provides a complex query mechanism. MongoDB needs to run on a 64-bit server, and it is best to deploy it separately, because it is a database, it also needs to be processed in hot and cold standby.

A high level overview of an application using MongoDB.

Node.js is nothing but a C++ program that runs on the application node.

We can Control Node.js using V8 JavaScript.

Any application your write using node.js will be written in JavaScript.

Node.js communicates to MongoDB using driver which contains a library of API’s for communicating with MongoDB.

Querying MONGODB:

Mongoimport utility is used to import data into MongoDB.

Loading example data: Product.json

Command: Mongoimport — db pcat — collection products < products.json

Selecting your data:

db.products.find() –will give you only limited rows.

db.products.count() — count of rows.

To select only one field (column)

db.products.find({},{name:1})

Selection one record. (top 1)

db.products.findOne()

To have a readable output:

db.products.find().toArray() — will load everything into the memory causing performance issues.

db.products.find().pretty() — will not load into memory.

To limit the number of records listed.

db.products.find().limit(5).toArray()

To skip N number of rows.

db.products.find({ }).limit(4).skip(2) — { } is nothing but the where clause.

Removing the id column.

db.products.find({},{name:1,_id:0})

To select more than one field.

db.products.find({},{name:1,brand:1})

Selecting based on the primary key Objectid.

db.products.findOne( {“_id”:ObjectId(“507d95d5719dbef170f15bfe”)})

selective columns with where clause.

db.products.find( {_id:”ac3"},{name:1,price:1})

db.products.find( {price:200},{name:1,price:1})

For Help: Help

to list the databases: show dbs

–Local database is an internal database used for activities like replication, sharding etc

To list collections: show collections

Operators in MongoDB: Query Selectors

Name

Description

$gt

Matches values that are greater than the value specified in the query. db.products.find( {price:{$gte:200}},{name:1,price:1})

$gte

Matches values that are equal to or greater than the value specified in the query. db.products.find( {price:{$gte:200}},{name:1,price:1})

$in

Matches any of the values that exist in an array specified in the query.

db.inventory.find( { qty: { $in: [ 5, 15 ] } } )

$lt

Matches values that are less than the value specified in the query. db.products.find( {price:{$gte:200}},{name:1,price:1})

$lte

Matches values that are less than or equal to the value specified in the query. db.products.find( {price:{$lte:200}},{name:1,price:1})

$ne

Matches all values that are not equal to the value specified in the query. db.inventory.find( { qty: { $ne: 20 } } )

$nin

Matches values that do not exist in an array specified to the query.

db.inventory.find( { qty: { $nin: [ 5, 15 ] } } )

$or

Joins query clauses with a logical OR returns all documents that match the conditions of either clause.

$and

Joins query clauses with a logical AND returns all documents that match the conditions of both clauses.

$not

Inverts the effect of a query expression and returns documents that do not match the query expression.

$nor

Joins query clauses with a logical NOR returns all documents that fail to match both clauses.

Name

Description

$exists

Matches documents that have the specified field.

$type

Selects documents if a field is of the specified type.

Evaluation Query Operators

Name

Description

$mod

Performs a modulo operation on the value of a field and selects documents with a specified result.

$regex

Selects documents where values match a specified regular expression.

$where

Matches documents that satisfy a JavaScript expression.

Name

Description

$geoWithin

Selects geometries within a bounding GeoJSON geometry.

$geoIntersects

Selects geometries that intersect with a GeoJSON geometry.

$near

Returns geospatial objects in proximity to a point.

$nearSphere

Returns geospatial objects in proximity to a point on a sphere.

Update Operators

Name

Description

$inc

Increments the value of the field by the specified amount.

$rename

Renames a field.

$setOnInsert

Sets the value of a field upon document creation during an upsert. Has no effect on update operations that modify existing documents.

$set

Sets the value of a field in an existing document.

$unset

Removes the specified field from an existing document.

Sort Command:

Db.collectionname.find({where,},{columns,}).sort()

Example:

db.products.find( {},{name:1,price:1}).sort({price:1}) — ascending

db.products.find( {},{name:1,price:1}).sort({price:-1}) — descending

db.products.find({price:{$exists:true}},{name:1,price:1}).sort({price:-1})

null is always less than 0

Cursors in MongoDB

Cursors:

for(var i=0;i<20000;i++) {db.test.insert({x:i,y:”hi”});}

For help:

Db.products.find().help() — cursor methods

db.test.find({x:19000})

{ “_id” : ObjectId(“53352d4024e9897b01cb2d58”), “x” : 19000, “y” : “hi” }

var cursor=db.test.find();

cursor.hasNext() — True

cursor.next()

{ “_id” : ObjectId(“53352d3f24e9897b01cae320”), “x” : 0, “y” : “hi” }

To select till hasNext() is false,

var cursor=db.test.find();while(cursor.hasNext())printjson(cursor.next());

To select only one column,

var cursor=db.test.find();while(cursor.hasNext())printjson(cursor.next().x);

Insert in MongoDB

Insert statement: Db.temperature.insert({x:3,y:4})

x={name:”Bluetooth Headset 509",

type: [“accessory”,”headset”],

price:30}

db.products.insert(x)

To find errors after a command executed.

db.getLastError()

Description of getLastError.

db.getLastErrorObj()

To view the index.

db.products.getIndexes()

Example of duplicate key error:

Db.products.insert({_id:”ac3”})

Notes: Bulk insert is also present in MongoDB.

Update in MongoDB

Two types of updates.

− Full document update.

db.test.update( { }, { “_id” : 100, “x” : “hello world”, y:123 })

− Partial update.

db.test.update( { _id : 100}, { “_id” : 100, “x” : “hello world”, y:123 })

$set

$push

$addtoset

$pop

$unset

Reference:

http://docs.mongodb.org/manual/reference/method/db.collection.update/

UPSERT statement.

Db.test.update({_id:”/sports/football”},{$inc:{views:1}},true)

Remove or Delete in MongoDB

Delete statement: db.test.remove({_id:100})

Delete all records: db.test.remove({})

Expressional delete: db.test2.remove({x:/ello/})

Hierarchical Delete:

{

_id: ObjectId(“508”),

name: “Jane”,

likes: [“tennis”,”golf”],

registered: false,

affr: {

city: “Lyon”,

country: “France”

}

}

db.users.remove({“addr.city”:”Lyon”,registered:false})

Common Commands in MongoDB

Instance Type: db.runCommand({isMaster:1}] Or db.runCommand(“isMaster”)

To look at help: db.isMaster

other helps: db.help()

ServerStatus: db.serverStatus()

what is currently running in the server: db.currentOp()

To kill a opid: db.killOp()

Create index: db.products.ensureIndex({name:1})

Running a query and getting the execution plan:

db.products.find({name:”AC3 Case Red”})

db.products.find({name:”AC3 Case Red”}).explain()

To list indexes: db.products.getIndexes() — Indexes are b-tree in MongoDB

we can also query the system catalog to query the indexes: db.system.indexes.find()

help on a collection: db.products.help()

To Drop an index: db.products.dropIndex({name:1})

To drop a collection: Db.products.drop()

To delete all the data in a collection: Db.products.remove({})

Namespace catalog: db.system.namespaces.find()

To list the databases info: db.runCommand({listDatabases:1})

Schema design principles

1. Need to pay attention to MongoDB’s own characteristics

To clearly understand the basic data unit of MongoDB. There are data tables with columns and rows in a relational database. The basic unit of MongoDB data is a BSON document. There are keys pointing to indefinite type values ​​in key values. MongoDB has instant query, but does not support join operations. Simple key-value storage can only obtain values ​​based on a single key, and does not support transactions. , But supports multiple atomic update operations.

2. Need to pay attention to the read and write characteristics of the system itself

For example, what is the read/write ratio, what kind of query is required, how the data is updated, will there be any concurrency problems, and whether the degree of data structure is high or low. The needs of the system itself determine mysql or MongoDB

3. Pay attention to the MongoDB schema design pattern

(1) Embedding and reference

When the child object always appears in the context of the parent object, use the embedded document; otherwise, store the child object in a separate collection

(2) One-to-many relationship

Add id to the dependent id in the “many” set relationship

(3) Many-to-many

Use an object array to point to another object in one of the correspondences

(4) The tree is

a concrete path, each node in the tree contains a path field, which specifically stores the id of each node’s ancestor

(5) Dynamic attributes

You can add indexes for different dynamic attributes. If you need to enclose the attributes in a range, you can use the key-value method, and then add an index on the unified key

(6) About transactions.

If you need transaction support, you can only choose another database, or provide compensatory transactions to solve the transaction problem

Pay attention to some principles in the design of the schema, such as

· Cannot create useless indexes

· Cannot store different types in the same field

· Cannot put multiple types of entities in a collection Cannot create large, deeply nested documents

· Do not create too many collections. The namespaces of collections, indexes, and databases are all limited

· Cannot create collections that cannot be sharded

4. Pay attention to some specific details in MongoDB

(1) Concerning the concept of

database Databases are logical and physical groupings of collections. MongoDB does not provide a grammar for creating databases. The database can only be created when the collection is inserted. After the database is created, a set of data files will be allocated on the disk. All collections, indexes, and other metadata of the database are stored in these files. You can check the disk status of the database through

db.stats()

(2) Focus on the concept of a

collection. A collection is a container for documents that are structurally or conceptually similar. The name of a collection can contain numbers, letters or symbols, but it must start with a letter or number and is complete.

The limited set name cannot exceed 128 characters. In fact, the symbol is very useful in the set. It can provide some kind of virtual namespace. This is an organizational principle and is treated the same as other sets. Can be used in the collection.

system.namespaces

system.indexes

(3) Focus on the document The

second is the key value. All strings in MongoDB are of UTF-8 type. The number types include double, int, and long. Date types are in UTC format, so the time seen in MongoDB will be 8 hours slower than Beijing time. The entire document size will be limited to 16m, because this can prevent the creation of unsightly data types, and small documents can improve performance. The ideal number range for batch insertion of documents is 10~200, and the size cannot exceed 16MB.

Index and query optimization

1. The rule of thumb for indexing

(1) Indexes can significantly reduce the workload required to obtain documents. Specific comparisons can be made through

.explain()

Method to compare

(2) When parsing the query, MongoDB selects an index for query through the optimal plan. When there is no most suitable index, it will use each index for query first, and finally select an optimal index for query.

(3) If there is a composite index of ab, then only the index for a is redundant

(4) The order of the keys in the composite index is very important

2. Index type

(1) Single-key index

(2) Composite index

(3) Unique index

(4) Sparse index

If the indexed field will have a null value, or a large number of documents do not contain the indexed key

3. Index construction problem

If the data set is large, it will take a long time to build the index and affect the performance of the program.

db.currentOp() //View the build time of the index

When mongorestore is used, the index will be rebuilt. When you have performed a large-scale deletion, you can use

db.values.reIndex()

• 1

Compress and rebuild the index

4. Identify slow queries

(1) Check the slow query log

grep -E ‘([0–9])+ms’ mongod.log //Use the grep command to identify command information

db.setProfillingLevel(2) //Use the planer to record every read and write to the log

db.setProfillingLevel(1) //Only record slow (100ms) operations

(2) Analyze slow queries

db.values.find({}).sort({close:-1}).limit(1).explain()

• The scanOrder field indicates that no index is used

• When the cursor is not indexed, BasicCursor is used, and BtreeCursor is used when indexing

• n represents the result set that needs to be returned

• nscanned indicates the number of documents that need to be traversed indexBounds indicates the index boundary

Note that the explain method of the new version of MongoDB requires parameters, otherwise only ordinary information is displayed

MongoDB replica set

This section also mainly presents the simplicity of MongoDB replica set construction, the robustness of the replica set, and the ease of monitoring

1. Why use replica sets

Provide master-slave replication capability, hot backup capability, failover capability

2. Construction method

rs.initiate()

rs.add(“localhost:40001”)

rs.add(“localhost:40002”,{arbiterOnly:true})

3. Monitoring

db.isMasrter()

rs.status()

4. How the replica set works

In fact, MongoDB’s operations on replica sets are similar to mysql master-slave operations. Let’s take a look at the mysql master-slave data flow process.

binlog -> from relay.log -> From bin.log -> From database

The log file that MongoDB mainly relies on is oplog

form oplog ->from oplog

The write operation is recorded first and added to the oplog of the master node. At the same time, all oplogs are copied from the node. First, check the timestamp of the last entry in your oplog; secondly, query all entries in the master node oplog that are greater than this timestamp; finally, add those entries to your oplog and apply them to your own library. The slave node uses long polling to immediately apply the new entry from the master node oplog.

When the following conditions are encountered, the slave node will stop replication

• If the slave node cannot find the point it is synchronizing in the oplog of the master node, it will permanently stop the replication

• Once a slave node fails to find its synchronized point in the oplog of the master node, it can no longer guarantee a perfect copy of the slave node

The local database saves all replica set element data and oplog logs

• replset.minvalid contains the initialization synchronization information of the specified replica set member

• system.replset is saved in the replica set configuration file

• system.indexes standard index description container

• me slaves are mainly used for writing concerns

You can use the following command to view the replication situation

db.oplog.rs.findOne()

• ts saves the BSON timestamp of the entry

• t is the description from the epoch

• i is the counter

• op means opcode

• ns indicates the relevant namespace

5. Heartbeat detection

Each member of the replica set pings all other members every second, and you can use rs.status() to see the node’s last heartbeat detection timestamp and health status.

6. Failover

There is no need to describe this point too much, but there is a special scenario. If both the slave node and the arbitration node are killed and only the master node is left, he will demote himself to the slave node.

7. Commit and rollback

If the data of the master node has not been written to the slave library, then the data cannot be considered as submitted. When the master node becomes a slave node, a rollback will be triggered. Those data that have not been written to the slave library will be deleted. You can use rollback The BSON file in the subdirectory restores the content of the rollback.

8. Drive and copy

(1) The single-node link

can only be linked to the master node. If it is linked to the slave node, the write operation will be rejected, but if the security mode is not used, because of the fire and forget feature of mongo, the write rejection will be abnormal Eat it.

(2) Using the replica set method to link

can automatically failover according to the write situation, but when the replica set performs a new election, it will still fail. If the safe mode is not used, it will still fail to write, but the reality is successful Case.

(3) Write attention

You can use write attention to pay attention to whether data has been written to the MongoDB library. Using write attention will consume performance, and you need to make a trade-off between speed and durability.

Sharding

Sharding is a conceptual realization of database sharding. Here is a brief summary of why sharding is used and the principles and operations of sharding.

1. Why sharding is needed

When the amount of data is too large, the memory occupied by the index and the working data set will increase, so it is necessary to solve this problem through sharding load

2. The working principle of sharding

(1) Sharding components

• Shard: Each shard is a replica set

• mongos router: is a router that directs read and write requests to the appropriate shards

• Configuration server config: persistent metadata of sharded clusters, including: global cluster configuration; each database, collection and data location of a specific range; a change record, which saves the historical information of data migration between shards. Configuration servers do not exist in the form of replica sets. Mongos submits information to the configuration server in two stages to ensure consistency between configuration servers.

(2) The core operation of

sharding Sharding a collection: sharding is divided according to the range of an attribute, MongoDB uses the so-called sharding key to let each document find its own position in these ranges

Block: a continuous segment key range located in a shard, which can be understood as several blocks to form a shard, and the shards constitute all the data of MongoDB

(3) Splitting and migration

Block splitting: there is only one block at initialization, reaching the maximum block size of 64MB or 100,000 documents will trigger the block splitting. Divide the original range into two, so there are two blocks, each with the same number of documents.

Migration: When the size of the data in the shards is not the same, the migration will occur. For example, if there is more data in shard A, some blocks in shard A will be transferred to shard B. The shard cluster achieves balance by moving blocks in the shards. It is managed by a software process called the equalizer. The task is to ensure that the data is evenly distributed among the shards. When the cluster has the most blocks and owns the block When the block difference of the least fragment is greater than 8, the equalizer will initiate an equalization process.

3. Sharding combat

Start two replica sets, three configuration servers, and a mongos process

Configure sharding

sh.help() //View help about sharding

sh.addShard() //Add shard

db,getSiblingDB(“config”).shards.find() //View the list of shards

sh.status()

sh.enableSharding(“cloud-docs”) //Open a shard on a database

db.getSiblingDB(“config”).databases,find() //View database list

sh.shardCollection(“cloud-docs.spreadsheets”,{username:1,_id:1}) //Use a shard key to define a shard set of spreadsheets, which are segmented according to the user name

sh.getSiiblingDB(“config”).collections.findOne() //View collection list

db.chunks.count() //View the number of blocks

db.chunks.findOne() //View block information

db.changelog.count(}what:”split”|) //View block information

db.changelog.find({what:”moveChunk.commit”}).count() //View log migration records

4. Fragmented query and index

(1) Fragment query type

• Targeted query: query contains shard key

• Global query or scatter/gather query: the query does not contain the shard key

• Query process: The query is routed to the specified shard through the shard key. Once it reaches a certain shard, the shard decides which index to use to execute the query.

(2) Index

Each shard maintains its own index. When an index is declared on the shard collection, each shard will build an independent index for its part of the collection. The shard collection on each shard should be Have the same index.

Sharded collections only allow unique indexes to be added to the _id field and the shard key, not elsewhere, because this requires communication between shards, which is complicated to implement.

When creating a shard, an index is created based on the shard key.

5. Select the shard key

(1) The shard key is unmodifiable, and the choice of the shard key is very important

(2) Inefficient shard key

• Poor distribution: If the BSON object ID is used, then all newly inserted documents will fall into a small continuous range and cannot be inserted scattered

• Lack of locality: Ascending shard keys have a clear direction, and completely random shard keys have no direction at all. The former cannot be inserted scattered, the latter inserted scattered, such as using MD5 as the shard key

(3) Ideal sharding key

• Distribute the inserted data evenly to each shard

• Ensure that CRUD operations can use locality to have sufficient granularity for block splitting

• The shard key that meets these requirements is usually composed of two fields, the first is coarse-grained, and the second is fine-grained

6. Sharding in the production environment

(1) Deployment topology

• Copy mongod: requires a separate deployment server

• Configuration server: configuration server does not need to have its own machine

According to different data centers

(2) Minimum requirements

· Each member of the replica set, whether it is a complete replica set node or an arbitration node, needs to be placed on a different machine. Each replica set member used for replication needs to have its own machine

· The replica set arbitration node is very lightweight and can share a machine with other processes

· The configuration server can also choose to share a machine with other processes

(3) Precautions

MongoDB simple sharding manual

In order to be able to setup mongodb simple sharding on your local machine you need to know the following:

Sharding consists of 3 parts

· config server — a simple mongo database which contains the configuration of the sharding

· mongos — an interaction point of application and cluster

· shard(s) — actual data storages

So in order to build sharding on your local you need to do the following steps:

1. Start the config server by running the following command with appropriate configuration db path (separate from shard db path)

mongod — dbpath ${directory_for_config_database} — port 37017

2. Start mongos process by specifying as a parameter configuration db url

mongos — configdb localhost:37017 — port 27017

3. Start shards on different ports and dbpath locations



mongod — dbpath ${directory_for_shard1} — port 47017

mongod — dbpath ${directory_for_shard2} — port 47018

mongod — dbpath ${directory_for_shard3} — port 47019

4. All the shards are managed by mongos, so the shards should not be accessed from application or changed in any way manually.

Now in order to connect shards to the mongos that has been started you need to do the following:

Start mongo, use admin database and run commands to connect shard to mongos:

mongo localhost:27017/admin

db.runCommand({“addShard”:”localhost:47017"});

db.runCommand({“addShard”:”localhost:47018"});

db.runCommand({“addShard”:”localhost:47019"});

you should see the following output in case of successful add:

{ “shardAdded” : “shard0000”, “ok” : 1 }

{ “shardAdded” : “shard0001”, “ok” : 1 }

{ “shardAdded” : “shard0002”, “ok” : 1 }

5. Now you have sharding in place, and you need to specify which database should be sharded and shard keys for collections.

In order to do that you need to run the following command on admin database of mongos:

mongo localhost:27017/admin

db.adminCommand({“enableSharding”:”be2"});

and e.g. to add users collection to the shard with key {emailAddress, branchId} you need to run:

db.adminCommand({“shardCollection” : “be2.user”, key : {“emailAddress” : 1, “branchId” : 1}})

So now you have configured 3 shards with one sharded collection. The entry point for application from now on should be the mongos — localhost:27017 in our example.

Deployment and management

1. Deployment

(1) Deployment architecture The

use of 64-bit machines and 32-bit machines will restrict the memory of mongodb to a maximum of 1.5GB

(2) CPU

mongodb will encounter the CPU bottleneck only when the index and working set can be put into the memory. The role of the CPU in mongodb is to retrieve data. If you see the saturation of CPU usage, you can query Slow query log, check if it is caused by query problem, if it can be solved by adding index,

mongodb will use CPU when writing data, but mongodb writing time only uses one core at a time, if there is frequent writing Behavior, this problem can be solved by sharding

(3) Memory

Large memory is the guarantee of mongodb. If the working set size exceeds the memory, it will cause performance degradation, because this will increase the action of loading data into memory

(4) The hard disk

mongodb will be forced to synchronize with the disk every 60s by default, which is called background refresh, which will generate I/O operations. When restarting, mongodb will load the data in the disk into the memory, and the high-speed disk will reduce the synchronization time

(5) File system

Use ext4 and xfs file system to

disable the last access time

vim /etc/fstab

(6) File Descriptor The

default file descriptor of Linux is 1024, which requires a large amount to increase this amount

(7) Clock

mongodb uses ntp server between each node server

2. Security

(1)Use the -bind_ip command when binding IP

(2)

Use the -auth command at startup

db.addUser(“”,””,true)

(3) Replica set identity authentication

Use keyFile, pay attention to the permission of the keyFile file must be 600, otherwise it will not start

3. Data import and export

mongoimport

mongoexport

4. Server configuration

(1) Topological structure

At least two nodes are required to build a replica set, and the arbitration node does not need to have its own server

(2) Journaling log When

writing data, the log is written first, and the data at this time is not directly written to the hard disk, but written to the memory.

However, the Journaling log consumes memory, so it can be closed on the main library and on the slave library. Startup

You can use a solid state drive for Journaling alone.

When inserting, you can use the driver to ensure that the Journaling is inserted and then feedback, but it will greatly affect performance

5. Log

The logpath option specifies the log storage address

-vvvvv option (the more v, the more detailed the output)

db.runCommand({logrotare:1}) Enable rolling log

6. Database monitoring commands

(1) serverStatus

globalLock represents the total time the server spends on write locks.

mem shows how to use memory

bits. Indicates the bit length of this machine.

resident represents the amount of physical memory occupied.

virtual represents the virtual memory used.

(2) top

(3) db.currentOp()

mongostat

Dynamic display of mongodb activity data

web console

Occupy the port 1000 up from the current mongodb listening port

Backup and restore

(1) mongodump exports

the contents of the database into BSON files, and mongorestore can read and restore these files

(2) mongorestore restores

the exported BSON file to the database

(3) Back up the original data file. This

can be done, but before the operation needs to be locked database processing db.runCommand({fsync:1,lock:true})

db.$cmd.sys.unlock.findOne() Request unlock operation, But the database will not be unlocked immediately, you need to use db.currentOp() to verify

Compression and repair

(1) Repair

mongd — repair Repair all databases

db.runCommand({repairDatabase:1}) Repair a single database

Repair is to read and rewrite all data files according to Jourling files and rebuild each index

(2) Compact

db.spreadsheets.reIndex()

db.runCommand({compact:”spreadsheets”})

Compression will rewrite the data file and rebuild all indexes of the collection. It needs to be shut down or run on the slave library. If you need to run on the main library, you need to add the force parameter to ensure that the write lock is added.

Performance tuning

(1) Monitor disk status

iostat

(2) To improve performance, check indexes and queries.

In general, scan as few documents as possible to

ensure that there are no redundant indexes. Redundant indexes will take up disk space and consume more memory. Need to do more

(3) Add memory

db.stats()

dataSize data size and indexSize index size, if the sum of the two is greater than the memory, it will affect performance

storageSize exceeds dataSize data size more than twice , it will affect performance due to disk fragmentation, need to be compressed

Thanks for reading this, if you like please subscribe me to get updated on this.