Package: m2bk
m2bkSend your mongodump backups straight to AWS S3m2bkis command line tool that performs a number ofmongodb database backups via mongodump, compresses them into a
gzipped tarball and finally sends them to anAWS S3 bucket(more options are about to be available).RequirementsContributingInstallationBasic usageOptionsConfiguration fileSections and directivesfs sectionmongodb sectionmongodb.host_defaults sectionmongodb.hosts sectionDrivers (driver section)dummys3DonatingCopyright and licensingRequirementspython>= 3.3boto>= 2.33envoy>= 0.0.3pyyaml>= 3.11mongodb>= 2.4clint>= 0.4.1ContributingThere are many ways in which you can contribute to m2bk.
Code patches are just one thing amongst others that you can submit to help the project.
We also welcome feedback, bug reports, feature requests, documentation improvements,
advertisement and testing.Feedback contributionsThis is by far the easiest way to contribute something.
If you’re using m2bk for your own benefit, don’t hesitate sharing.
Feel free tosubmit issues and enhancement requests.Code contributionsCode contributions (patches, new features) are the most obvious way to help with the project’s development.
Since this is so common we ask you to follow our workflow to most efficiently work with us.
For code contributions, we follow the “fork-and-pull” Git workflow.1. Fork, then clone your repo on GitHubgit clone git@github.com:your-username/m2bk.git
git add origin upstream https://github.com/axltxl/m2bk.gitIf you already forked the repo, then be sure to merge
the most recent changes from “upstream” before making a pull request.git pull upstream2. Create a new feature branch in your local repogit checkout -b my_feature_branch3. Make your changes, then make sure the tests passespyvenv m2bk-pyve && source m2bk-pyve/bin/activate
python3 setup.py test4. Commit your changes once donegit commit -a -m "My commit message"
git push origin my_feature_branchSubmit apull requestwith your feature branch containing your changes.InstallationInstallation of m2bk can be made directly from source, viapiporeasy_install, whichever you prefer.Option # 1: pip$ pip install m2bkOption # 2: from source$ git clone git@github.com:axltxl/m2bk.git
$ cd m2bk
$ python3 setup.py installOption # 3: easy_install$ easy_install m2bkFrom this point you can edit yourconfiguration file$ vi /etc/m2bk/m2bk.yamlBasic UsageNormal execution$ m2bkQuiet output$ m2bk -qDry run$ m2bk -dSpecify an alternate configuration file$ m2bk -c /path/to/my/custom/m2bk.yamlOptionsm2bk [options]--versionshow version number and exit-h|--helpshow a help message and exit-c[file] |--config=[file]|--config[file]specify configuration file to use-d|--dry-rundon’t actually do anything-q|--quietquiet output--ll|--log-level=[num]set logging output level-lLOG_FILE |--log-fileLOG_FILE set log fileConfiguration fileThe configuration is handled through a simpleYAMLfile including a series ofsections(which are YAML objects), each one
composed bydirectives(YAML numbers, strings or arrays), these will
determine a corresponding behavior onm2bk. Ifm2bkdoes not receive
any configuration file on command line, it will try to read/etc/m2bk.yaml.Please note the configuration format is still a work in progress and will most likely change in the early stages of m2bk.The following is an example of what a configuration file looks like:---
driver:
  name: s3
  options:
    aws_access_key_id: "SDF73HSDF3663KSKDJ"
    aws_secret_access_key: "d577273ff885c3f84dadb8578bb41399"
fs:
  output_dir: "/opt/tmp/mydir"
mongodb:
  mongodump: "/opt/bin/mongodump"
  host_defaults:
    port: 666
    user_name: "satan"
    password: "14mh4x0r"
  hosts:
    foo:
      address: "foo.example.local"
      port: 34127
      dbs:
        - "app"
        - "sessions"
        - "another_one"
    bar:
      address: "bar.example.com"
      password: "1AmAn07h3rh4x0r"
      auth_db: bar
      dbs:
          - customers
          - sessionsThrough this configuration file, you can set key variables about the
databases you want to backup and the AWS S3 bucket you wish to send them
to.Configuration file: sections and directivesfssectionThis section has directives regarding files and directories manipulationDirectivesfs.output_dirType:stringDefault value :/tmp/m2bkRole: directory where m2bk is going to temporarily save backup filesmongodbsectionThis section holds directives regarding mongodb serversm2bkis going
to connect to, including databases that are going to be backed up throughmongodump.Example:mongodb:
    mongodump: "/opt/bin/mongodump"
    host_defaults:
        user_name: tom
        address: db.example.local
        password: "457893mnfs3j"
        dbs:
          - halloran
          - grady
    hosts:
        foo:
            address: db0.example.internal
            port: 27654
            user_name: matt
            password: "myS3cr37P455w0rd"
            dbs:
              # This list is going to be merged with dbs at host_defaults, thus
              # the resulting dbs will be:
              # ['halloran', 'grady', 'jack', 'wendy', 'danny']
              - jack
              - wendy
              - danny
        bar: {} # This one is going to acquire all host_defaults values
        host_with_mixed_values:
            # This host will inherit port, password and dbs from host_defaults
            address: moloko.example.internal
            user_name: alex
            address: localhost
            auth_db: milk_plusDirectivesmongodb.mongodumpType:stringDefault value :mongodumpRole: full path to themongodumpexecutable used by m2bkmongodb.host_defaultssectionMany directives (such as user name and/or password) could be common
among the databases that are going to be backed up. For this reason, it
is best to simply put those common directives under a single section,
this is entirely optional but also it is the best for easily manageable
configuration files in order to avoid redundancy, the supported
directives areuser_name,password,port,dbsandauth_db.
Seehostssection.mongodb.hostssectionThis is an object/hash, where each element contains a series of
directives relative to a mongodb database located at a server, its
specifications and databases themselves held by it, these are
the main values used bymongodumpwhen it does its magic. For each
entry inside thehostssection, these are its valid directives:Directivesmongodb.hosts.*.addressType:stringRequired: YESRole: mongodb server locationmongodb.hosts.*.portType:integerRequired: NODefault value :mongo.host_defaults.port | 27017Role: mongodb server listening portmongodb.hosts.*.user_nameType:stringRequired: NODefault value :mongodb.host_defaults.user_name | m2bkRole: user name used for authentication against the mongodb servermongodb.hosts.*.passwordType:stringRequired: NODefault value :mongodb.host_defaults.pass | "pass"Role: password used for authentication against the mongodb servermongodb.hosts.*.auth_dbType:stringRequired: NODefault value :adminRole: authentication databasemongodb.hosts.*.dbsType:arrayRequired: NODefault value :mongodb.host_defaults.dbs | []Role: a list of databases who are expected inside the mongodb serverNOTE: particular “dbs” on one host will be merged with those of “host_defaults”Drivers (driversection)Once backup files have been generated, they are then handled by a driver, whose
job is to transfer resulting backup files to some form of storage (depending
on the driver set on configuration). Drivers (and their options) are
set and configured inside thedriversection like so:driver:
    # First of all, you need to tell m2bk which driver to use
    name: dummy

    # Inside this key, driver options are set
    options:
      hello: world
      another_option: another_valuePer driver, there are a bunch of availableoptionsto tweak them.
These options vary among drivers. Though there is only one driver available on
m2bk, there will be more drivers available with new releases. Current available
drivers are the following:dummyThis driver is just a placeholder for testing out the driver interface as
it won’t do a thing on backup files.OptionsThere are no options for this driver. Any option passed to this driver
will be logged at debug level.s3This driver holds directives regarding AWS credentials thatm2bkis going to use in order to upload themongodumpbackups to S3.
If eitheraws_access_key_idoraws_secret_access_keyare not specified,
this driver will not try to use them to authenticate against AWS and will rely
onboto configfor that matter.Example:driver:
    name: s3
    options:
        aws_access_key_id": "HAS6NBASD8787SD"
        aws_secret_access_key: "d41d8cd98f00b204e9800998ecf8427e"
        s3_bucket: "mybucket"Optionsaws_access_key_idType:stringRequired: NORole: AWS access key IDaws_secret_access_keyType:stringRequired: NORole: AWS access key IDs3_bucketType:stringRequired: NODefault value:m2bkRole: name of the main S3 bucket where m2bk is going to upload the compressed backups for each mongodb server specified inmongodbsectionDonatingShow your love and support this project viagratipayCopyright and LicensingCopyright (c) Alejandro RicoveriPermission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the
“Software”), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:The above copyright notice and this permission notice shall be included
in all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.