Repository: StackStorm/st2

Issue:
"NotUniqueError: Tried to save duplicate unique keys" on st2 register content
> 

Getting the following error on Ubuntu 16 latest `stable` st2 packages during the `st2ctl --register-all` (first & clean run):

```
    Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
    2017-11-10 16:04:14,401 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
    2017-11-10 16:04:20,355 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.update" })
    2017-11-10 16:04:20,432 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.value_change:99914b932bd37a50b983c5e7c90ae93b" })
    2017-11-10 16:04:20,451 ERROR [-] Conflict while trying to save in DB.
    Traceback (most recent call last):
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
        model_object = cls._get_impl().add_or_update(model_object)
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 336, in add_or_update
        instance.save()
      File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
        raise NotUniqueError(message % six.text_type(err))
    NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
    2017-11-10 16:04:20,567 INFO [-] =========================================================
    2017-11-10 16:04:20,568 INFO [-] ############## Registering triggers #####################
    2017-11-10 16:04:20,568 INFO [-] =========================================================
    2017-11-10 16:04:20,673 INFO [-] Registered 0 triggers.
```

Happens randomly, probably some race and I couldn't reproduce it consistently so far.
Hence, still good to leave it here until we or someone else finds more info.


Discussion:
Seeing it again on StackStorm first install.
![screenshot-from-2018-03-06-15-32-37](https://user-images.githubusercontent.com/1533818/37035207-71ba38d2-2154-11e8-8d00-c2708186c5ca.gif)


The worst thing is that the error doesn't result in `st2ctl reload` script failure and so installation goes further as nothing wrong happened.

I think it should fail on failure like this, so we stop the provision on that error instead of going further. @Kami do you think that's something at minimum we can fix?
More of this: 
```
    virtualbox-iso: 20180320T114415+0000 Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
    virtualbox-iso: 20180320T114418+0000 2018-03-20 11:44:18,975 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
    virtualbox-iso: 20180320T114421+0000 2018-03-20 11:44:21,985 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114421+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114421+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114421+0000     instance.save()
    virtualbox-iso: 20180320T114421+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.generic.actiontrigger" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,039 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.generic.actiontrigger:99914b932bd37a50b983c5e7c90ae93b" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,146 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.generic.notifytrigger" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,547 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.value_change:99914b932bd37a50b983c5e7c90ae93b" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,586 ERROR [-] Conflict while trying to save in DB.
    virtualbox-iso: 20180320T114422+0000 Traceback (most recent call last):
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/persistence/base.py", line 173, in add_or_update
    virtualbox-iso: 20180320T114422+0000     model_object = cls._get_impl().add_or_update(model_object)
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 388, in add_or_update
    virtualbox-iso: 20180320T114422+0000     instance.save()
    virtualbox-iso: 20180320T114422+0000   File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/mongoengine/document.py", line 364, in save
    virtualbox-iso: 20180320T114422+0000     raise NotUniqueError(message % six.text_type(err))
    virtualbox-iso: 20180320T114422+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,762 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,762 INFO [-] ############## Registering triggers #####################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,763 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] Registered 0 triggers.
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] ############## Registering sensors ######################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,887 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] Registered 1 sensors.
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] =========================================================
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] ############## Registering runners ######################
    virtualbox-iso: 20180320T114422+0000 2018-03-20 11:44:22,934 INFO [-] =========================================================
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] Registered 14 runners.
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] =========================================================
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] ############## Registering actions ######################
    virtualbox-iso: 20180320T114423+0000 2018-03-20 11:44:23,537 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] Registered 51 actions.
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] ############## Registering rules ########################
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,921 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] Registered 1 rules.
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] =========================================================
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] ############## Registering aliases ######################
    virtualbox-iso: 20180320T114424+0000 2018-03-20 11:44:24,958 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:24,999 INFO [-] Registered 4 aliases.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] ############## Registering policy types #################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,000 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] Registered 3 policy types.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] ############## Registering policies #####################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,023 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] Registered 0 policies.
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] ############## Registering configs ######################
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,024 INFO [-] =========================================================
    virtualbox-iso: 20180320T114425+0000 2018-03-20 11:44:25,026 INFO [-] Registered 0 configs.
```
I am also facing same issue 

```
20190405T104945+0000 Registering content...[flags = --config-file /etc/st2/st2.conf --register-all]
20190405T104949+0000 2019-04-05 10:49:49,621 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
20190405T104949+0000 2019-04-05 10:49:49,626 INFO [-] Successfully connected to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
20190405T104958+0000 2019-04-05 10:49:58,815 ERROR [-] Conflict while trying to save in DB.
20190405T104958+0000 Traceback (most recent call last):
20190405T104958+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T104959+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T104959+0000     instance.save(validate=validate)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T104959+0000     raise NotUniqueError(message % six.text_type(err))
20190405T104959+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.key_value_pair.update:99914b932bd37a50b983c5e7c90ae93b" })
20190405T104959+0000 2019-04-05 10:49:58,853 ERROR [-] Conflict while trying to save in DB.
20190405T104959+0000 Traceback (most recent call last):
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T104959+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T104959+0000     instance.save(validate=validate)
20190405T104959+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T104959+0000     raise NotUniqueError(message % six.text_type(err))
20190405T104959+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.value_change" })
20190405T104959+0000 2019-04-05 10:49:58,910 ERROR [-] Conflict while trying to save in DB.
20190405T105000+0000 Traceback (most recent call last):
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105000+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105000+0000     instance.save(validate=validate)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105000+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105000+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.key_value_pair.delete" })
20190405T105000+0000 2019-04-05 10:49:59,090 ERROR [-] Conflict while trying to save in DB.
20190405T105000+0000 Traceback (most recent call last):
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105000+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105000+0000     instance.save(validate=validate)
20190405T105000+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105000+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105000+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.sensor.process_spawn:99914b932bd37a50b983c5e7c90ae93b" })
20190405T105001+0000 2019-04-05 10:49:59,133 ERROR [-] Conflict while trying to save in DB.
20190405T105001+0000 Traceback (most recent call last):
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105001+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105001+0000     instance.save(validate=validate)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105001+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105001+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_type_d_b index: uid_1 dup key: { : "trigger_type:core:st2.sensor.process_exit" })
20190405T105001+0000 2019-04-05 10:49:59,199 ERROR [-] Conflict while trying to save in DB.
20190405T105001+0000 Traceback (most recent call last):
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/persistence/base.py", line 174, in add_or_update
20190405T105001+0000     model_object = cls._get_impl().add_or_update(model_object, validate=True)
20190405T105001+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/models/db/__init__.py", line 452, in add_or_update
20190405T105001+0000     instance.save(validate=validate)
20190405T105002+0000   File "/opt/stackstorm/st2/lib/python2.7/site-packages/mongoengine/document.py", line 403, in save
20190405T105002+0000     raise NotUniqueError(message % six.text_type(err))
20190405T105002+0000 NotUniqueError: Tried to save duplicate unique keys (E11000 duplicate key error collection: st2.trigger_d_b index: uid_1 dup key: { : "trigger:core:st2.sensor.process_exit:99914b932bd37a50b983c5e7c90ae93b" })
```
I fixed it by 
1. uninstalling `st2* , postgres* and mongo*`  packages 
2. Delete all data directory for postgres and mongo
3. Delete the configs related with these packages
3. Reinstall packages

Also st2chatops 2.10 is dependent on nodejs > nodejs-8.0, which doesn't get install with the single command installation process, so I installed it from `https://rpm.nodesource.com/pub_8.x/el/7/x86_64/nodejs-8.1.0-1nodesource.el7.centos.x86_64.rpm`
I can confirm i'm seeing this too, as reported in #4626.

Here's a gist of my install log: https://gist.github.com/nmaludy/875acebf3772d10384df5eccfc2605e9

I noticed that this only happened when installing on CentOS 7, i couldn't reproduce using Ubuntu 16.04.

Here are my steps to reproduce:

``` shell
git clone https://github.com/StackStorm/st2vagrant
cd st2vagrant
BOX=centos/7 vagrant up
```
I will try to have a look this week.

From a quick glance - some of those errors (trigger type already exists) are not fatal.

IIRC, we already made some changes in the past to log those errors under DEBUG instead of ERROR so it doesn't confuse the user, but it may need more work (and there might be another face hiding there which is fatal).
I was able to reproduce this issue today.

It's indeed related to a non-fatal race - we try to register internal trigger types on each service startup and during register content script run.

Operation itself is idempotent so failure isn't fatal (it simply means internal trigger has already been registered by another service). Only problem is with logging - we log error as fatal, even though it's not.

I will open a PR which fixes the logging and logs this error under ``DEBUG`` instead of ``ERROR``.

Resolving patch:
diff --git a/st2common/st2common/services/triggers.py b/st2common/st2common/services/triggers.py
--- a/st2common/st2common/services/triggers.py
+++ b/st2common/st2common/services/triggers.py
@@ -245,7 +245,7 @@ def create_trigger_db(trigger_api):
     return trigger_db
 
 
-def create_or_update_trigger_db(trigger):
+def create_or_update_trigger_db(trigger, log_not_unique_error_as_debug=False):
     """
     Create a new TriggerDB model if one doesn't exist yet or update existing
     one.
@@ -269,7 +269,8 @@ def create_or_update_trigger_db(trigger):
     if is_update:
         trigger_db.id = existing_trigger_db.id
 
-    trigger_db = Trigger.add_or_update(trigger_db)
+    trigger_db = Trigger.add_or_update(trigger_db,
+        log_not_unique_error_as_debug=log_not_unique_error_as_debug)
 
     extra = {'trigger_db': trigger_db}
 
@@ -331,13 +332,20 @@ def cleanup_trigger_db_for_rule(rule_db):
     Trigger.delete_if_unreferenced(existing_trigger_db)
 
 
-def create_trigger_type_db(trigger_type):
+def create_trigger_type_db(trigger_type, log_not_unique_error_as_debug=False):
     """
     Creates a trigger type db object in the db given trigger_type definition as dict.
 
     :param trigger_type: Trigger type model.
     :type trigger_type: ``dict``
 
+    :param log_not_unique_error_as_debug: True to lot NotUnique errors under debug instead of
+                                          error log level. This is to be used in scenarios where
+                                          failure is non-fatal (e.g. when services register
+                                          internal trigger types which is an idempotent
+                                          operation).
+    :type log_not_unique_error_as_debug: ``bool``
+
     :rtype: ``object``
     """
     trigger_type_api = TriggerTypeAPI(**trigger_type)
@@ -349,13 +357,23 @@ def create_trigger_type_db(trigger_type):
     if not trigger_type_db:
         trigger_type_db = TriggerTypeAPI.to_model(trigger_type_api)
         LOG.debug('verified trigger and formulated TriggerDB=%s', trigger_type_db)
-        trigger_type_db = TriggerType.add_or_update(trigger_type_db)
+        trigger_type_db = TriggerType.add_or_update(trigger_type_db,
+            log_not_unique_error_as_debug=log_not_unique_error_as_debug)
+
     return trigger_type_db
 
 
-def create_shadow_trigger(trigger_type_db):
+def create_shadow_trigger(trigger_type_db, log_not_unique_error_as_debug=False):
     """
     Create a shadow trigger for TriggerType with no parameters.
+
+    :param log_not_unique_error_as_debug: True to lot NotUnique errors under debug instead of
+                                          error log level. This is to be used in scenarios where
+                                          failure is non-fatal (e.g. when services register
+                                          internal trigger types which is an idempotent
+                                          operation).
+    :type log_not_unique_error_as_debug: ``bool``
+
     """
     trigger_type_ref = trigger_type_db.get_reference().ref
 
@@ -368,16 +386,24 @@ def create_shadow_trigger(trigger_type_db):
                'type': trigger_type_ref,
                'parameters': {}}
 
-    return create_or_update_trigger_db(trigger)
+    return create_or_update_trigger_db(trigger,
+                                       log_not_unique_error_as_debug=log_not_unique_error_as_debug)
 
 
-def create_or_update_trigger_type_db(trigger_type):
+def create_or_update_trigger_type_db(trigger_type, log_not_unique_error_as_debug=False):
     """
     Create or update a trigger type db object in the db given trigger_type definition as dict.
 
     :param trigger_type: Trigger type model.
     :type trigger_type: ``dict``
 
+    :param log_not_unique_error_as_debug: True to lot NotUnique errors under debug instead of
+                                          error log level. This is to be used in scenarios where
+                                          failure is non-fatal (e.g. when services register
+                                          internal trigger types which is an idempotent
+                                          operation).
+    :type log_not_unique_error_as_debug: ``bool``
+
     :rtype: ``object``
     """
     assert isinstance(trigger_type, dict)
@@ -399,7 +425,8 @@ def create_or_update_trigger_type_db(trigger_type):
         trigger_type_api.id = existing_trigger_type_db.id
 
     try:
-        trigger_type_db = TriggerType.add_or_update(trigger_type_api)
+        trigger_type_db = TriggerType.add_or_update(trigger_type_api,
+            log_not_unique_error_as_debug=log_not_unique_error_as_debug)
     except StackStormDBObjectConflictError:
         # Operation is idempotent and trigger could have already been created by
         # another process. Ignore object already exists because it simply means
diff --git a/st2common/st2common/triggers.py b/st2common/st2common/triggers.py
--- a/st2common/st2common/triggers.py
+++ b/st2common/st2common/triggers.py
@@ -14,6 +14,7 @@
 # limitations under the License.
 
 from __future__ import absolute_import
+
 import six
 
 from mongoengine import ValidationError
@@ -23,7 +24,8 @@
 from st2common import log as logging
 from st2common.constants.triggers import (INTERNAL_TRIGGER_TYPES, ACTION_SENSOR_TRIGGER)
 from st2common.exceptions.db import StackStormDBObjectConflictError
-from st2common.services.triggers import create_trigger_type_db, create_shadow_trigger
+from st2common.services.triggers import create_trigger_type_db
+from st2common.services.triggers import create_shadow_trigger
 from st2common.services.triggers import get_trigger_type_db
 from st2common.models.system.common import ResourceReference
 
@@ -36,11 +38,12 @@
 
 def _register_internal_trigger_type(trigger_definition):
     try:
-        trigger_type_db = create_trigger_type_db(trigger_type=trigger_definition)
+        trigger_type_db = create_trigger_type_db(trigger_type=trigger_definition,
+                                                 log_not_unique_error_as_debug=True)
     except (NotUniqueError, StackStormDBObjectConflictError):
         # We ignore conflict error since this operation is idempotent and race is not an issue
-        LOG.debug('Internal trigger type "%s" already exists, ignoring...' %
-                  (trigger_definition['name']), exc_info=True)
+        LOG.debug('Internal trigger type "%s" already exists, ignoring error...' %
+                  (trigger_definition['name']))
 
         ref = ResourceReference.to_string_reference(name=trigger_definition['name'],
                                                     pack=trigger_definition['pack'])
@@ -52,12 +55,13 @@ def _register_internal_trigger_type(trigger_definition):
     # trigger types with parameters do no require a shadow trigger.
     if trigger_type_db and not trigger_type_db.parameters_schema:
         try:
-            trigger_db = create_shadow_trigger(trigger_type_db)
+            trigger_db = create_shadow_trigger(trigger_type_db,
+                log_not_unique_error_as_debug=True)
 
             extra = {'trigger_db': trigger_db}
             LOG.audit('Trigger created for parameter-less internal TriggerType. Trigger.id=%s' %
                       (trigger_db.id), extra=extra)
-        except StackStormDBObjectConflictError:
+        except (NotUniqueError, StackStormDBObjectConflictError):
             LOG.debug('Shadow trigger "%s" already exists. Ignoring.',
                       trigger_type_db.get_reference().ref, exc_info=True)
 
@@ -73,7 +77,11 @@ def register_internal_trigger_types():
     """
     Register internal trigger types.
 
-    Note: This method blocks until all the trigger types have been registered.
+    NOTE 1: This method blocks until all the trigger types have been registered.
+
+    NOTE 2: We log "NotUniqueError" errors under debug and not error. Those errors are not fatal
+    because this operation is idempotent and NotUniqueError simply means internal trigger type
+    has already been registered by some other service.
     """
     action_sensor_enabled = cfg.CONF.action_sensor.enable
 


Test patch: