Repository: netbox-community/netbox

Issue:
deficiency in new webhook implementation
<!--
    Before opening a new issue, please search through the existing issues to
    see if your topic has already been addressed. Note that you may need to
    remove the "is:open" filter from the search bar to include closed issues.

    Check the appropriate type for your issue below by placing an x between the
    brackets. For assistance with installation issues, or for any other issues
    other than those listed below, please raise your topic for discussion on
    our mailing list:

        https://groups.google.com/forum/#!forum/netbox-discuss

    Please note that issues which do not fall under any of the below categories
    will be closed. Due to an excessive backlog of feature requests, we are
    not currently accepting any proposals which extend NetBox's feature scope.

    Do not prepend any sort of tag to your issue's title. An administrator will
    review your issue and assign labels as appropriate.
--->
### Issue type
[ ] Feature request <!-- An enhancement of existing functionality -->
[X] Bug report      <!-- Unexpected or erroneous behavior -->
[ ] Documentation   <!-- A modification to the documentation -->

<!--
    Please describe the environment in which you are running NetBox. (Be sure
    to verify that you are running the latest stable release of NetBox before
    submitting a bug report.) If you are submitting a bug report and have made
    any changes to the code base, please first validate that your bug can be
    recreated while running an official release.
-->
### Environment
* Python version:  python 2.7.5
* NetBox version:  develop-2.4

<!--
    BUG REPORTS must include:
        * A list of the steps needed for someone else to reproduce the bug
        * A description of the expected and observed behavior
        * Any relevant error messages (screenshots may also help)

    FEATURE REQUESTS must include:
        * A detailed description of the proposed functionality
        * A use case for the new feature
        * A rough description of any necessary changes to the database schema
        * Any relevant third-party libraries which would be needed
-->
### Description
Testing out the webhook implementation and discovered the following bug:
     when a model contains a custom field of type date the worker is unable to serialize the data for transmission

```
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/rq/worker.py", line 793, in perform_job
    rv = job.perform()
  File "/usr/lib/python2.7/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/usr/lib/python2.7/site-packages/rq/job.py", line 605, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/opt/netbox/netbox/extras/webhooks_worker.py", line 44, in process_webhook
    prepared_request = requests.Request(**params).prepare()
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 259, in prepare
    hooks=self.hooks,
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 307, in prepare
    self.prepare_body(data, files, json)
  File "/usr/lib/python2.7/site-packages/requests/models.py", line 427, in prepare_body
    body = json_dumps(json)
  File "/usr/lib64/python2.7/site-packages/simplejson/__init__.py", line 382, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 291, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 373, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python2.7/site-packages/simplejson/encoder.py", line 268, in default
    o.__class__.__name__)
TypeError: Object of type date is not JSON serializable
```


Discussion:
@ridoline I am unable to reproduce this with a Date custom field on the Device model. Please provide the exact steps necessary to raise this issue.
@lawpwins,  I suppose it might have been an issue with my data or has been resolved, as I recently refreshed my development box with production data and the issue is no longer manifesting itself.


I played with this some last night and was able to raise the issue a handful of times, but I have yet to find a root cause or even a reliable way of triggering the issue. I will be digging into this more.
I finally ran this down. There are two separate issues in play. First, date type custom field values are currently actual datetime objects and the default json serializer in the requests library doesn't handle this well. I am still a little unclear why DRF is okay with this, since we are using the same API serializers. Second, the current signaling system for Webhooks suffers from the same problems of related objects (custom fields and tags) not being associated at the time of the post_save signal, as the original Change Logging implementation. This was addresses in Change Logging in df1f339. I think it makes a lot of sense to wrap the Webhook signaling into the ChangeLoggingMiddleware since the mechanics of both features are very similar.
Just to be clear, this was broken out from #2282 and thus this issue still needs to be addressed.
I finally nailed the root cause down. DRF processes the JSON encoding after the serialization, so by manually using the serialisers we miss the DRF specific encoding which makes use of there own JSON encoder class which correctly handles datatime objects (and many others) https://github.com/encode/django-rest-framework/blob/7b1582e00e91001ec07cd394520ec5cdd2c2add1/rest_framework/utils/encoders.py#L34. Currently we are using the built-in JSON encoder of the requests library, so we simply need to specify an alternate encoder to fix this.

Resolving patch:
diff --git a/netbox/extras/webhooks_worker.py b/netbox/extras/webhooks_worker.py
--- a/netbox/extras/webhooks_worker.py
+++ b/netbox/extras/webhooks_worker.py
@@ -1,8 +1,10 @@
 import hashlib
 import hmac
-
 import requests
+import json
+
 from django_rq import job
+from rest_framework.utils.encoders import JSONEncoder
 
 from extras.constants import WEBHOOK_CT_JSON, WEBHOOK_CT_X_WWW_FORM_ENCODED, OBJECTCHANGE_ACTION_CHOICES
 
@@ -13,9 +15,9 @@ def process_webhook(webhook, data, model_class, event, timestamp):
     Make a POST request to the defined Webhook
     """
     payload = {
-        'event': dict(OBJECTCHANGE_ACTION_CHOICES)[event],
+        'event': dict(OBJECTCHANGE_ACTION_CHOICES)[event].lower(),
         'timestamp': timestamp,
-        'model': model_class.__name__,
+        'model': model_class._meta.model_name,
         'data': data
     }
     headers = {
@@ -28,7 +30,7 @@ def process_webhook(webhook, data, model_class, event, timestamp):
     }
 
     if webhook.http_content_type == WEBHOOK_CT_JSON:
-        params.update({'json': payload})
+        params.update({'data': json.dumps(payload, cls=JSONEncoder)})
     elif webhook.http_content_type == WEBHOOK_CT_X_WWW_FORM_ENCODED:
         params.update({'data': payload})
 


Test patch: